Crawl4AI with Docker: Turn Any Website into LLM-Ready Markdown (Linux, Windows, Mac)

What you get in one minute#

Crawl4AI is an open-source web crawler that opens a page in a real browser and returns clean, LLM-ready Markdown, so your RAG pipeline or AI agent can read websites without HTML clutter. You can run it as a Python library or as a Docker server with a small REST API.

Logo: the official Crawl4AI project (github.com/unclecode/crawl4ai), its attribution badge from the repository

Use it to feed documentation into a local chatbot, build a knowledge base from your own sites, or give an agent a "read this page" tool. You send POST /md with a URL and get Markdown back. A headless Chromium runs the page's JavaScript first.

At a glance:

• What it is: one container with an API, a browser pool (Playwright Chromium) and a built-in dashboard and playground.
• What you get back: Markdown (fit filtered or raw), cleaned HTML, links, media and, with a model connected, structured JSON.
• Secure by default: since 0.9.0 every endpoint except /health needs a token, and the server will not start in network mode without one.
• Robots.txt is opt-in: the check is off unless you turn it on. Step 5 shows how.
• Platforms: Linux, Windows and Mac through Docker Engine or Docker Desktop; the library needs Python 3.10 or newer.
• Privacy: pages are fetched from your own IP. Prompts go only to an LLM if you connect one.

How it fits together#

Your app calls one local port with a token. The API hands the job to a pooled browser, which loads the page; the generator turns the HTML into Markdown and returns it in the same HTTP reply.

Drawn for this guide from the official Compose file and docs at v0.9.5. Dashed parts are optional.

Library, server or cloud?#

Compared with Firecrawl: Crawl4AI is a single container under Apache 2.0 with a token required by default, and it also works as a Python library. Firecrawl is a multi-service stack under AGPL-3.0 with an open API by default.

Before you start#

The official docs name Docker 20.10.0+, docker compose v2.24+ and 4 GB RAM. Check your tools on Linux and Mac:

docker --version
docker compose version

On Windows (PowerShell) the same two commands work.

Checklist:

• [ ] docker compose version prints v2.24 or newer
• [ ] Port 11235 is free on your machine
• [ ] Docker Desktop users: at least 4 GB of memory in Docker Desktop's resource settings
• [ ] For the HTTPS section only: a domain with a DNS record for the server, and ports 80 and 443 free

Tested on 2026-10-31: Linux = not run; Windows = Docker Desktop with Linux containers (server) and Python 3.13 (library); Mac = not run. The Linux and Mac commands are the same as the Windows ones apart from the shell syntax, and come from the official docs.

Step 1: Create the folder and a token#

The server needs a secret token. Without it, the container only listens on its own inside loopback and your host cannot reach it. We keep everything in ~/docker_volumes/crawl4ai.

Linux#

mkdir -p ~/docker_volumes/crawl4ai/{data,backups}
cd ~/docker_volumes/crawl4ai
printf 'CRAWL4AI_API_TOKEN=%s\n' "$(openssl rand -hex 32)" > .env
chmod 600 .env

Windows (PowerShell)#

$dir = "$HOME\docker_volumes\crawl4ai"
"data", "backups" | ForEach-Object { New-Item -ItemType Directory -Force "$dir\$_" | Out-Null }
Set-Location $dir
$bytes = New-Object byte[] 32
[System.Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($bytes)
$token = ($bytes | ForEach-Object { $_.ToString('x2') }) -join ''
"CRAWL4AI_API_TOKEN=$token" | Set-Content -Path .env -Encoding ascii

Mac (Terminal)#

Same commands as Linux.

Check it worked. The file has one line with a 64-character token:

grep -c . .env

On Windows use (Get-Content .env).Length, which prints the character count of the line (it should be 83: the 18-character name, = and 64 characters). Never paste the token into chats, tickets or Git.

Step 2: Write the Compose file#

This file uses the pinned image unclecode/crawl4ai:0.9.5 and the hardening settings of the official docker-compose.yml at v0.9.5: all Linux capabilities dropped, a read-only filesystem with small temporary mounts, a 4 GB memory cap and a health check. Two changes of ours: the port is bound to 127.0.0.1 so only your own computer can reach it, and the token comes from .env. The official file clones the whole repository and builds; you do not need that to run the published image.

services:
  crawl4ai:
    image: unclecode/crawl4ai:0.9.5
    container_name: crawl4ai
    restart: unless-stopped
    ports:
      - "127.0.0.1:11235:11235"
    env_file:
      - path: .llm.env
        required: false
    environment:
      - CRAWL4AI_API_TOKEN=${CRAWL4AI_API_TOKEN:?Put CRAWL4AI_API_TOKEN in the .env file}
    shm_size: "1gb"
    user: "appuser"
    cap_drop:
      - ALL
    security_opt:
      - no-new-privileges:true
    read_only: true
    tmpfs:
      - /tmp
      - /var/lib/redis:uid=999,gid=999,mode=0700
      - /var/lib/crawl4ai/outputs:uid=999,gid=999,mode=0700
      - /home/appuser/.crawl4ai:uid=999,gid=999,mode=0700
      - /home/appuser/.cache/url_seeder:uid=999,gid=999,mode=0700
      - /home/appuser/.gunicorn:uid=999,gid=999,mode=0700
    deploy:
      resources:
        limits:
          memory: 4G
          pids: 512
        reservations:
          memory: 1G
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:11235/health"]
      interval: 30s
      timeout: 10s
      retries: 3
      start_period: 40s

Save it as compose.yaml in the folder from Step 1.

Linux#

cd ~/docker_volumes/crawl4ai
nano compose.yaml        # paste the YAML above, save
docker compose config --quiet && echo "compose file OK"

Windows (PowerShell)#

Set-Location "$HOME\docker_volumes\crawl4ai"
notepad compose.yaml     # paste the YAML above, save as compose.yaml (not .txt)
docker compose config --quiet; if ($?) { "compose file OK" }

Mac (Terminal)#

Same commands as Linux (use nano or open -e compose.yaml).

Check it worked. compose file OK appears and no error. The read_only and tmpfs lines are not decoration: the server writes its working files into those temporary mounts, and an older report (issue #2027) shows what happens when one is missing.

Tip: Do not copy the Compose file from an older blog post or from the docs of version 0.8 or 0.9.0. Issue #2091 shows a 0.9.2 file that Compose v5 rejected, and issue #2027 a 0.9.0 file that crash-looped. Use the file above or the one at the release tag.

Step 3: Start the server and check health#

Linux#

cd ~/docker_volumes/crawl4ai
docker compose up -d
docker compose ps
curl -s http://localhost:11235/health

Windows (PowerShell)#

Set-Location "$HOME\docker_volumes\crawl4ai"
docker compose up -d
docker compose ps
Invoke-RestMethod -Uri http://localhost:11235/health

Mac (Terminal)#

Same commands as Linux.

Check it worked. The first start downloads about 2.3 GB. Allow about 30 seconds after the container starts. Real output from our Windows run (columns trimmed):

NAME       IMAGE                      COMMAND                SERVICE    STATUS                    PORTS
crawl4ai   unclecode/crawl4ai:0.9.5   "bash entrypoint.sh"   crawl4ai   Up 28 seconds (healthy)   127.0.0.1:11235->11235/tcp
{"status":"ok","timestamp":1793427044.8106606,"version":"0.9.5"}

/health is the only endpoint that needs no token. If you get "Empty reply" or "connection reset" instead, read Troubleshooting: the token did not reach the container.

Open http://localhost:11235/playground in a browser for the interactive page and http://localhost:11235/dashboard for the monitor. Both have an "API token" bar at the top: paste your token and click Set, or every call answers "Authentication required".

Step 4: Turn a page into Markdown#

POST /md is the simplest endpoint. The request has url and an optional filter f:

Linux#

cd ~/docker_volumes/crawl4ai
set -a; . ./.env; set +a      # loads CRAWL4AI_API_TOKEN into this shell
curl -s -X POST http://localhost:11235/md \
  -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"url": "https://example.com"}' | jq -r .markdown | tee data/example.md

Windows (PowerShell)#

Set-Location "$HOME\docker_volumes\crawl4ai"
$token = (Get-Content .env | Where-Object { $_ -like 'CRAWL4AI_API_TOKEN=*' }) -replace '^CRAWL4AI_API_TOKEN=', ''
$h = @{ Authorization = "Bearer $token" }
$body = @{ url = "https://example.com" } | ConvertTo-Json
$r = Invoke-RestMethod -Method Post -Uri http://localhost:11235/md -Headers $h -ContentType "application/json" -Body $body
$r.markdown | Set-Content data\example.md
$r.success

Mac (Terminal)#

Same commands as Linux (brew install jq first if jq is missing).

Check it worked. The answer has "success": true, "filter": "fit" and the page as Markdown. Real output for example.com from our run (the page itself currently lists its text in several languages):

This domain is for use in documentation examples without needing permission. This is not a service; avoid relying on it for testing and monitoring purposes.
...
[Learn more](https://iana.org/help/example-domains)

Without the header the server answers HTTP 401 with {"detail": "Authentication required"} (we checked this too).

Tip: /md is for a single page. For several URLs, structured extraction or browser settings use POST /crawl, which takes urls, browser_config and crawler_config and returns a results list. Its markdown field is an object (raw_markdown, fit_markdown, references_markdown and more), not a string.

Step 5: Crawl politely and respect robots.txt#

The robots.txt check is off by default. The documented parameter check_robots_txt is False unless you set it, and /md has no field for it at all. For anything beyond your own sites, use /crawl with the check on. The server's built-in rate limiter also waits one to two seconds between requests by default (crawler.rate_limiter.base_delay).

Linux#

curl -s -X POST http://localhost:11235/crawl \
  -H "Authorization: Bearer $CRAWL4AI_API_TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"urls": ["https://example.com"], "crawler_config": {"type": "CrawlerRunConfig", "params": {"cache_mode": "bypass", "check_robots_txt": true}}}' \
  | jq '.results[0] | {success, status_code, error_message}'

Windows (PowerShell)#

$body = @{ urls = @("https://example.com"); crawler_config = @{ type = "CrawlerRunConfig"; params = @{ cache_mode = "bypass"; check_robots_txt = $true } } } | ConvertTo-Json -Depth 6
$c = Invoke-RestMethod -Method Post -Uri http://localhost:11235/crawl -Headers $h -ContentType "application/json" -Body $body
$c.results[0] | Select-Object success, status_code, error_message

Mac (Terminal)#

Same commands as Linux.

Check it worked. For a page the site allows, success is true and status_code is 200. For a page its robots.txt forbids, the crawl is refused with an error. Real output from our run for a URL that a large search site disallows (fields as the tool printed them):

success: False   status_code: 403   error_message: Access denied by robots.txt

Tip: Scraping is your responsibility. Read the site's terms, crawl your own sites first, keep volumes low and do not use the tool to get around bot protection. The library has stealth options; this guide does not teach them.

Step 6 (optional): Use the Python library instead#

If you only need a script, skip Docker. We install into a virtual environment so nothing else on your system changes, and pin the same version.

Linux#

python3 -m venv ~/crawl4ai-env
~/crawl4ai-env/bin/pip install -U "crawl4ai==0.9.5"
~/crawl4ai-env/bin/crawl4ai-setup
~/crawl4ai-env/bin/crawl4ai-doctor

Windows (PowerShell)#

python -m venv "$HOME\crawl4ai-env"
& "$HOME\crawl4ai-env\Scripts\pip.exe" install -U "crawl4ai==0.9.5"
& "$HOME\crawl4ai-env\Scripts\crawl4ai-setup.exe"
& "$HOME\crawl4ai-env\Scripts\crawl4ai-doctor.exe"

Mac (Terminal)#

Same commands as Linux.

crawl4ai-setup downloads the browsers: in our Windows run Chromium (197.5 MiB), a headless shell (118.1 MiB) and a second set for the "undetected" mode. If it fails on Linux, the README's fallback is python -m playwright install --with-deps chromium.

Save this as fetch.py. It turns a page into Markdown and honours robots.txt, with your own user agent:

import asyncio
from crawl4ai import AsyncWebCrawler, CrawlerRunConfig

async def main():
    config = CrawlerRunConfig(check_robots_txt=True, user_agent="MyBot/1.0 ([email protected])")
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://example.com", config=config)
        print(result.success, result.status_code)
        print(result.markdown[:300])

asyncio.run(main())

Run it with ~/crawl4ai-env/bin/python fetch.py (Windows: & "$HOME\crawl4ai-env\Scripts\python.exe" fetch.py; Mac like Linux).

Check it worked. crawl4ai-doctor ends with "Crawling test passed!", and fetch.py prints True 200 and the text. Since 0.9.5 the tools print a short Crawl4AI Cloud notice; CRAWL4AI_NO_CLOUD_NOTICE=1 hides it.

For structured data without an LLM use the CSS or XPath extraction strategies in the docs. To use a local model from the library, the README's example sets LLMConfig(provider="ollama/llama3.3", api_token="no-token"); our Ollama guide shows how to run one. We did not run an LLM through the server.

Put it behind HTTPS (with or without Traefik)#

Skip this section if only your own computer calls Crawl4AI. Every endpoint already needs the token; HTTPS stops it travelling in clear text. Both routes put the container on a shared Docker network called proxy. Create it once (same command in PowerShell and Terminal):

docker network create proxy

Then add this file next to compose.yaml. Compose reads compose.override.yaml automatically:

services:
  crawl4ai:
    networks:
      - proxy

networks:
  proxy:
    external: true

With Traefik#

On a server that already runs Traefik v3 with the Docker provider, add labels to the same override file. Our entrypoint is called https; use your own name.

services:
  crawl4ai:
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.crawl4ai.rule=Host(`crawl4ai.<your-domain>`)"
      - "traefik.http.routers.crawl4ai.entrypoints=https"
      - "traefik.http.routers.crawl4ai.tls=true"
      - "traefik.http.services.crawl4ai.loadbalancer.server.port=11235"

networks:
  proxy:
    external: true

Run docker compose up -d in the Crawl4AI folder.

Check it worked. curl -I https://crawl4ai.<your-domain>/health returns 200 over HTTPS.

Without Traefik: Caddy#

Choose this if you have no Traefik. Caddy gets and renews certificates itself. Use the first override file (no labels). Then create the Caddy folder:

mkdir -p ~/docker_volumes/caddy/{data,config}

On Windows: New-Item -ItemType Directory -Force "$HOME\docker_volumes\caddy\data", "$HOME\docker_volumes\caddy\config".

Save this as docker-compose.yml in that folder (caddy:2.11.7 is the latest Caddy release in October 2026):

services:
  caddy:
    image: caddy:2.11.7
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80/tcp"
      - "443:443/tcp"
      - "443:443/udp"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./data:/data
      - ./config:/config
    networks:
      - proxy

networks:
  proxy:
    external: true

And this as Caddyfile for a public domain (Let's Encrypt):

crawl4ai.example.com {
    reverse_proxy crawl4ai:11235
}

For a home-only name, add tls internal as the first line inside the block and use a name such as crawl4ai.home.arpa. Start Caddy with docker compose up -d in its folder (same in PowerShell and Terminal).

For tls internal, every device must trust Caddy's root certificate. Find and copy it out:

docker exec caddy find /data -name root.crt
docker cp caddy:<path-from-find> ./caddy-root.crt

Install it as a trusted root: Windows "Trusted Root Certification Authorities"; macOS Keychain Access with "Always Trust"; Linux your distribution's CA store. These OS steps are general practice, not from the Crawl4AI docs.

Check it worked. curl -I https://crawl4ai.example.com/health returns 200 (use curl.exe in Windows PowerShell). We did not run the HTTPS routes ourselves; they follow the Caddy and Traefik docs.

Tip: Let's Encrypt needs the DNS record to point at the server and ports 80 and 443 to reach it. A certificate error usually means the name in the Caddyfile differs from the DNS record.

Security and testing#

The official docs call 0.9.0 and newer "secure-by-default". These are the controls to keep:

Quick tests after each change (on Windows use the PowerShell lines from Step 3):

docker compose ps
docker compose logs --tail=100 crawl4ai
curl -s http://localhost:11235/health

What leaves your machine: page requests to the sites you crawl (from your IP), prompts to an LLM only if you connect one, and the image download from Docker Hub. We found no telemetry setting in the docs; the only extra message is the Cloud notice in the library tools. Release 0.9.5 fixed six security problems in the server, which is why we pin it rather than an older tag.

Using it for RAG and AI agents#

Crawl4AI is the "read the web" half of a RAG pipeline: fetch pages, store the Markdown, then chunk and embed it. Our Qdrant guide shows the storage half and the Docling guide does the same for PDFs.

The server also speaks MCP, so an agent such as Claude Code can use md, crawl and other tools directly. The self-hosting page shows claude mcp add --transport sse c4ai-sse http://localhost:11235/mcp/sse. Because every endpoint now needs a token and that example has none, add a header (--header "Authorization: Bearer <YOUR_TOKEN>"). We did not test this route; the docs say WebSocket clients that cannot set headers may pass ?token=....

Common mistakes#

These come from the official migration notes, the docs' own warnings and the issue tracker.

Tip: If a request "succeeds" but nothing happens, read the response body, not the status banner. The docs say the playground can show "Success" for an authentication error, and that output_path and old hook code are ignored silently.

Tip: Crawl4aiDockerClient from the Python SDK has no public setting for the 0.9 token (open issue #2474). Plain requests or httpx with an Authorization header always works.

Troubleshooting#

"can't set distinct values on 'pids_limit' and 'deploy.resources.limits.pids'"#

This is the error from issue #2091: an older Compose file with Compose v5. The file in Step 2 sets pids under deploy.resources.limits as the official 0.9.5 file does.

Known bugs#

• Issue #2202 (open, August 2026): on non-streaming /crawl, pages and renderer processes can pile up over weeks. Watch docker stats and restart the container now and then.
• Issue #2123 (open, August 2026): the memory guard can read the host's RAM when no container limit is set. Keep memory: 4G in the Compose file.
• Issue #2419 (open, October 2026): the server can deadlock when it recycles a browser after cancelled requests. A restart clears it.

Diagnostics: docker compose ps, docker compose logs --tail=200 crawl4ai, docker inspect crawl4ai, docker stats crawl4ai.

Updates and backups#

Update. Read the changelog and the migration guide first, then change the tag in compose.yaml and recreate (same commands in PowerShell, after Set-Location "$HOME\docker_volumes\crawl4ai"):

cd ~/docker_volumes/crawl4ai
docker compose pull
docker compose up -d

To roll back, put the old tag back and run the same two commands. The library: pip install -U "crawl4ai==<new>" and crawl4ai-setup again.

Back up. The server keeps no data you need: results come back in the HTTP reply, and the in-container job store is temporary (task results expire after one hour by default). What you must keep is your configuration:

cd ~/docker_volumes/crawl4ai
tar -czf "backups/crawl4ai-config-$(date +%F).tar.gz" compose.yaml .env data $(ls .llm.env compose.override.yaml config 2>/dev/null)
chmod 600 backups/crawl4ai-config-*.tar.gz

Windows (PowerShell):

Set-Location "$HOME\docker_volumes\crawl4ai"
$files = @("compose.yaml", ".env", "data") + @(Get-ChildItem .llm.env, compose.override.yaml -ErrorAction SilentlyContinue | ForEach-Object Name)
Compress-Archive -Path $files -DestinationPath "backups\crawl4ai-config-$(Get-Date -Format yyyy-MM-dd).zip" -Force

Mac: same commands as Linux. The archive contains your token, so keep it private.

Schedule. Linux and Mac: a crontab line for 03:00, 0 3 * * * cd ~/docker_volumes/crawl4ai && tar -czf backups/crawl4ai-config-$(date +\%F).tar.gz compose.yaml .env data (cron needs the \%). Windows: a daily 03:00 task in Task Scheduler that runs the PowerShell lines above.

Restore test. Unpack into an empty folder and compare:

mkdir -p /tmp/c4ai-restore && tar -xzf ~/docker_volumes/crawl4ai/backups/crawl4ai-config-<date>.tar.gz -C /tmp/c4ai-restore
diff /tmp/c4ai-restore/compose.yaml ~/docker_volumes/crawl4ai/compose.yaml && echo "backup matches"

On Windows, use Expand-Archive into a new folder and Compare-Object (Get-Content <file1>) (Get-Content <file2>).

Performance and resources#

The docs ask for at least 4 GB RAM for the container, and the Compose file caps it at 4 GB with 1 GB reserved. We measured, on a Windows PC with Docker Desktop: 2.28 GB download, 8.18 GB on disk, healthy about 30 seconds after start, and 325 to 487 MiB with 168 processes after two single-page requests.

To change config.yml, mount your own copy. The docs warn your file completely replaces the default, so start from the official one:

cd ~/docker_volumes/crawl4ai
mkdir -p config
curl -sL -o config/config.yml https://raw.githubusercontent.com/unclecode/crawl4ai/v0.9.5/deploy/docker/config.yml

On Windows use Invoke-WebRequest -Uri <same URL> -OutFile config\config.yml. Edit the values, add - ./config/config.yml:/app/config.yml:ro under volumes: in the Compose file, and run docker compose up -d. Monitor with docker stats crawl4ai and the /dashboard page.

FAQ#

Is Crawl4AI free? Yes. The code is Apache 2.0 with an attribution requirement (see Sources). The hosted Crawl4AI Cloud is a separate paid service that you never need for this guide.

Do I need a GPU? No. The default image has no PyTorch. A GPU only helps a model you connect yourself, such as Ollama.

Does it work on Apple Silicon or a Raspberry Pi? The official image covers linux/amd64 and linux/arm64 (README). We did not test either.

Can it crawl a whole site? Yes, with deep crawling (breadth-first, depth-first or best-first). Over the API you must enable it with CRAWL4AI_ALLOW_DEEP_CRAWL=true, and the server clamps it to 100 pages and depth 5 unless you change config.yml. In the library it is available without those limits. Keep max_pages low on sites you do not own.

Sources#

• Crawl4AI self-hosting guide and docs home - read October 2026 (the page still names 0.9.2 as the latest release; the repository shows 0.9.5)
• README, docker-compose.yml, MIGRATION.md, config.yml, CHANGELOG.md and parameters reference at tag v0.9.5 - read October 2026
• Release v0.9.5, Docker Hub: unclecode/crawl4ai and PyPI: crawl4ai - read October 2026
• LICENSE - Apache 2.0 with an attribution appendix
• Caddy release v2.11.7 and Caddy docs - read October 2026
• GitHub issues #2026, #2027, #2064, #2091, #2123, #2202, #2211, #2308, #2419, #2474 - read October 2026

Attribution required by the project's license: "This product includes software developed by UncleCode (https://x.com/unclecode) as part of the Crawl4AI project (https://github.com/unclecode/crawl4ai)." Read the LICENSE yourself for your own use; this guide is not legal advice.

Tested by DIY Lab Hub on 2026-10-31 (server and library on Windows with Docker Desktop; not run on Linux or Mac).

Checked against the official docs on 2026-10-31.