vLLM with Docker: a fast OpenAI-compatible API on your own NVIDIA GPU (Linux, Windows, Mac)

What you get in one minute#

vLLM is an open-source inference server that turns a model from Hugging Face into a fast, OpenAI-compatible API on your own NVIDIA GPU, and the official Docker image makes it a one-container job.

Your apps keep using the OpenAI SDK or plain curl; only the base URL changes to http://127.0.0.1:8000/v1. vLLM handles many requests at once, so people pick it to run an API service and tune GPU use. It can reuse repeated prompt starts (prefix caching) and split a model over several GPUs, but neither is a magic speed-up.

One honest warning first: CUDA only exists on NVIDIA GPUs. Linux works directly. Windows works through WSL2. A Mac cannot run this Docker image with GPU speed, and the section for Mac users below says what to do instead.

Why v0.32.0 and not an older release? We checked the issue tracker for the last four weeks. There are several bug reports, but they are tied to specific setups (one very-long-context model, ROCm, video input, one workstation GPU). Nobody reports a general install failure or lost data, and some of the reported problems also hit older versions. So the guide uses the current release. If your model matches one of those reports, test first and read the release notes.

How it fits together#

Your app sends an OpenAI-style request to the vLLM container, the container runs the model on your GPU, and the model files come from a cache folder that is filled from Hugging Face on the first start.

Not sure which route is yours? This picture answers it in three questions.

Before you start#

You need an NVIDIA GPU with compute capability 7.5 or higher, a current NVIDIA driver, and Docker with Compose.

There is no universal VRAM minimum: weights, context length and parallel requests all use memory. This guide starts with Qwen/Qwen3-0.6B, the small model from the official Docker example. You do not need the CUDA toolkit on the host, only the driver and the container toolkit.

Check the tools:

docker --version
docker compose version
nvidia-smi

• [ ] nvidia-smi shows your GPU and a driver version (R580 or newer for the default image)
• [ ] Port 8000 is free on the machine
• [ ] Any old vLLM folder is backed up
• [ ] About 10 GB or more free disk space for the image and the model cache (more for bigger models)
• [ ] Only for the Traefik section: a domain, DNS and ports 80/443

Tested on: not run by DIY Lab Hub for this guide (Linux = not run, Windows = not run, Mac = not run). The commands were checked against the official vLLM, NVIDIA and Hugging Face docs on 2026-10-26 (see Sources).

Step 1: Make Docker see your GPU#

Docker must be able to run nvidia-smi inside a container before vLLM has any chance.

If nvidia-smi already fails on the host, stop here and fix the driver first. The container toolkit cannot repair a broken driver.

Linux#

These commands are for Ubuntu or Debian with Docker Engine already installed. They follow the NVIDIA install guide, which currently pins toolkit 1.20.1-1.

sudo apt-get update
sudo apt-get install -y --no-install-recommends ca-certificates curl gnupg2

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
  nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
  libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

The nvidia-ctk line edits Docker's daemon settings, and the restart stops running containers for a moment. Other Linux distributions use other package commands. See the NVIDIA install guide in Sources.

Now test with the vLLM image itself (it replaces the server with nvidia-smi):

docker pull vllm/vllm-openai:v0.32.0
docker run --rm --runtime nvidia --gpus all --entrypoint nvidia-smi vllm/vllm-openai:v0.32.0

Windows (PowerShell)#

On Windows you run vLLM inside WSL2 (Ubuntu). Open PowerShell as administrator:

wsl --install -d Ubuntu
wsl --update
wsl --list --verbose

Restart Windows if it asks. Then:

1. Install a current NVIDIA driver on Windows (the WSL2 GPU support comes with it). Do not install a Linux NVIDIA driver inside WSL.
2. Install Docker Desktop, turn on the WSL2 engine and enable integration for your Ubuntu distribution.
3. Use Linux containers.

Docker Desktop manages its own daemon, so do not run the Linux nvidia-ctk commands there. Test the GPU in PowerShell:

docker run --rm --gpus all --entrypoint nvidia-smi vllm/vllm-openai:v0.32.0

From Step 2 on, run the Linux commands inside the Ubuntu shell (wsl -d Ubuntu), so your files live on the Linux side.

Mac (Terminal)#

There is nothing to install for NVIDIA, because a Mac has no CUDA. Do not expect Docker Desktop on a Mac to pass the Apple GPU into this image. Your options:

• Use the Mac as the client. Run vLLM on a Linux or Windows machine with an NVIDIA GPU and call it from the Mac with the curl commands in Step 4.
• vLLM-Metal. The vLLM docs describe a community-maintained plugin that uses Apple Metal and MLX on Apple Silicon (macOS Sonoma or later), with MLX-optimised models. It is a native install, not this Docker image.
• vLLM CPU build. The vLLM docs list Apple Silicon for the CPU backend. It works, but it is not GPU speed.
• Ollama or llama.cpp. Common local choices on a Mac. See our Ollama guide.

Check it worked: the container prints a table with your GPU name and driver version. Expected output looks like the normal nvidia-smi table, listing your own card.

Potential problem: could not select device driver "" with capabilities: [[gpu]] means the toolkit is not configured. Repeat the nvidia-ctk line and restart Docker.

Step 2: Create the folders and the secrets#

Make the folders under ~/docker_volumes/vllm and a .env file with your API key and, if you need one, a Hugging Face token.

HF_TOKEN only lets vLLM download gated or private models from Hugging Face. VLLM_API_KEY is the key your own apps send to your server. Never give apps your Hugging Face token.

Linux#

mkdir -p ~/docker_volumes/vllm/{data/hf-cache,config,logs,backups}
cd ~/docker_volumes/vllm
printf 'VLLM_API_KEY=%s\nHF_TOKEN=\n' "$(openssl rand -hex 32)" > .env
chmod 600 .env

If a model needs a Hugging Face token, put it after HF_TOKEN= with an editor.

Windows (PowerShell)#

Run this in the Ubuntu WSL shell (open it with wsl -d Ubuntu in PowerShell). The commands are the same as Linux:

mkdir -p ~/docker_volumes/vllm/{data/hf-cache,config,logs,backups}
cd ~/docker_volumes/vllm
printf 'VLLM_API_KEY=%s\nHF_TOKEN=\n' "$(openssl rand -hex 32)" > .env
chmod 600 .env

Mac (Terminal)#

On a Mac you only need the key to call a remote server. Ask the server's owner for the key, or keep it in a shell variable:

read -rsp "vLLM API key: " VLLM_API_KEY; export VLLM_API_KEY; echo

Check it worked: ls -l .env shows -rw------- and the file has two lines. Never commit it or paste it anywhere.

Step 3: Write the Compose file and start vLLM#

One service, one pinned image, the GPU reserved, the model cache on disk and the port bound to loopback.

Linux#

Create docker-compose.yml in ~/docker_volumes/vllm. It has no version: line.

services:
  vllm:
    image: vllm/vllm-openai:v0.32.0
    container_name: vllm
    restart: unless-stopped
    ipc: host
    env_file:
      - .env
    ports:
      - "127.0.0.1:8000:8000"
    volumes:
      - ./data/hf-cache:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command:
      - --model
      - Qwen/Qwen3-0.6B
      - --host
      - 0.0.0.0
      - --port
      - "8000"
      - --served-model-name
      - local-chat
      - --max-model-len
      - "4096"
      - --gpu-memory-utilization
      - "0.85"
      - --max-num-seqs
      - "4"

What the settings mean:

• ipc: host is what the official example uses (--ipc=host); without shared memory you can see bus or NCCL errors.
• VLLM_API_KEY from .env turns on key checking (the documented variable for --api-key).
• The port is published on 127.0.0.1 only, while inside the container vLLM listens on all addresses.
• --served-model-name local-chat is the name your apps use, whatever model you load.
• --max-model-len 4096 caps the context; start small. --gpu-memory-utilization 0.85 is below the default 0.92 for headroom, and --max-num-seqs 4 limits parallel requests.
• Prefix caching is on by default, so no flag is needed.

Start it and watch the log:

docker compose config
docker compose up -d
docker compose logs -f vllm

The first start downloads the model and prepares the engine, so give it a few minutes. Press Ctrl+C to leave the log; the container keeps running.

Check it worked: the log ends with a line saying the server is running on port 8000, and this answers with 200:

curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/health

Potential problem: if the container exits at once, read docker compose logs vllm and look at the Troubleshooting table.

Windows (PowerShell)#

Run everything above inside the Ubuntu shell (wsl -d Ubuntu). The file and the commands are identical, and the GPU comes from Docker Desktop.

Mac (Terminal)#

Nothing to start on a Mac, because there is no CUDA. Point your client at the GPU machine instead.

Step 4: Test the API#

List the models, then send one chat request with your key.

Linux#

set -a; . ~/docker_volumes/vllm/.env; set +a
curl -s http://127.0.0.1:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"local-chat","messages":[{"role":"user","content":"Explain what an inference API does in two sentences."}],"max_tokens":256}'

Windows (PowerShell)#

The server runs in WSL2 and Docker Desktop, and 127.0.0.1:8000 is normally reachable from Windows too. Paste your key in the first line:

$env:VLLM_API_KEY = "<YOUR_VLLM_API_KEY>"
$h = @{ Authorization = "Bearer $env:VLLM_API_KEY" }
Invoke-RestMethod http://127.0.0.1:8000/v1/models -Headers $h | ConvertTo-Json -Depth 5
$body = @{ model = "local-chat"; max_tokens = 256; messages = @(@{ role = "user"; content = "Explain what an inference API does in two sentences." }) } | ConvertTo-Json -Depth 6
Invoke-RestMethod -Method Post http://127.0.0.1:8000/v1/chat/completions -Headers $h -ContentType "application/json" -Body $body | ConvertTo-Json -Depth 8

If you want the real curl, call curl.exe, because plain curl in PowerShell is an alias.

Mac (Terminal)#

Same commands as Linux. For a remote server, replace 127.0.0.1 with the server's address (and use HTTPS, see the Traefik section):

curl -s http://127.0.0.1:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"
curl -s http://127.0.0.1:8000/v1/chat/completions \
  -H "Authorization: Bearer $VLLM_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"local-chat","messages":[{"role":"user","content":"Explain what an inference API does in two sentences."}],"max_tokens":256}'

Check it worked: the first call lists local-chat in a data array. The second returns the usual OpenAI-style JSON with the answer in choices[0].message.content. Without the key you get a 401 error. Qwen3 can write its thinking before the answer, which is normal for that model.

Any OpenAI SDK works too: set the base URL to http://127.0.0.1:8000/v1, the key to your VLLM_API_KEY, and the model to local-chat. "OpenAI-compatible" means the requests look the same. It does not mean the local model answers like a hosted one, so test your app with the model you pick. If you want one address for several backends, see our LiteLLM guide.

Put it behind HTTPS with Traefik#

On a server with a domain, let Traefik publish only the /v1 routes over HTTPS and remove the host port.

This is for a machine with a domain. Local-only users can skip it. It assumes Traefik v3 with the Docker provider, an entrypoint called https and a certificate for your hostname. Create the network once if it is missing: docker network create proxy. In docker-compose.yml, delete the ports: block, then add:

services:
  vllm:
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.vllm.rule=Host(`llm.<your-domain>`) && PathPrefix(`/v1/`)"
      - "traefik.http.routers.vllm.entrypoints=https"
      - "traefik.http.routers.vllm.tls=true"
      - "traefik.http.services.vllm.loadbalancer.server.port=8000"
networks:
  proxy:
    external: true

Merge this into your existing file. Add traefik.http.routers.vllm.tls.certresolver=<your-resolver> if you use ACME. The PathPrefix(/v1/) rule is on purpose: /metrics and /health are not covered by the API key (see below), so they stay off the public route. HTTPS also adds no rate limiting. Add that in Traefik if the server is shared.

Check it worked: curl -I https://llm.<your-domain>/v1/models answers 401 without a key, and curl -s -o /dev/null -w '%{http_code}\n' https://llm.<your-domain>/metrics does not return your metrics.

Security and testing#

The API key protects the model routes, not every route, so keep the port private.

• Keep .env at chmod 600, out of git. Use a separate VLLM_API_KEY, never your Hugging Face token.
• vLLM's security docs say --api-key does not protect every endpoint, and /metrics and /health are among those you should not assume are protected. Keep the host port on 127.0.0.1, or publish only /v1/ through a gateway.
• On a shared server, read the vLLM security page on prefix-cache sharing (cache_salt), and add rate limits at the gateway, because huge requests can tie up your GPU.
• Pin the image tag. Do not run latest on a server people depend on.

Quick tests after every change:

docker compose ps
docker compose logs --tail=100 vllm
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:8000/health

Check /health, not just docker compose ps. A report on GitHub shows the API process staying alive while the engine behind it had died, with /health returning 503. A running container is not proof that the model works.

Which settings matter#

Almost every startup problem is one of three things: the model does not fit, the context is too long, or the driver is too old.

Quantisation: AWQ and GPTQ need a checkpoint already made in that format, and FP8 computes natively only on compute capability 8.9 or higher. Version 0.31.0 changed the online FP8 setting, so old recipes with --quantization fp8 may fail. Get the basic setup running first.

Troubleshooting#

Read the first error in docker compose logs vllm before you change anything.

Exact error text changes between versions, so match on the key words. "The NVIDIA driver on your system is too old" and "the provided PTX was compiled with an unsupported toolchain" both mean the image's CUDA is newer than the host driver supports: update the driver. "Failed to infer device type" means vLLM cannot find a working GPU: check Step 1, and for one run add VLLM_LOGGING_LEVEL=DEBUG under environment: (remove it afterwards, it can slow the server). "NCCL error: unhandled system error" is often shared memory: keep ipc: host and do not switch on privileged mode just to try.

Known bugs#

Reports on the vLLM GitHub issues page in the four weeks before this guide (versions around 0.32.0) include:

• A very long prompt (about 262,000 tokens) on one new model crashed the engine when speculative decoding and CUDA graphs were on. --enforce-eager avoided it in the report.
• Garbage output with the FlashInfer backend on 0.29.0 and 0.32.0 alike.
• A startup deadlock on one workstation GPU with a 70B NVFP4 model.

None of these hit the small starter setup in this guide. If you serve a model that matches one, try an earlier release for comparison and read the release notes. Version 0.32.0 also narrowed the default CORS origins and removed some old options, so old launch scripts may need edits.

Diagnostic commands:

docker logs vllm --tail 100
docker compose ps
docker inspect vllm
nvidia-smi

Updates and backups#

Update by changing the pinned tag on purpose, and keep what you need to roll back.

1. Read the new release notes: security fixes, breaking changes, CUDA variants, model support.
2. Write down the tag that works now (v0.32.0).
3. Change image: in docker-compose.yml to the new tag.
4. Pull, recreate, and run the Step 4 tests again.

cd ~/docker_volumes/vllm
docker compose pull vllm
docker compose up -d vllm
docker compose logs -f vllm

To roll back, put the old tag back and run docker compose up -d again.

What to back up: the Compose file, config, image tag, model name and your secrets. The model cache is optional because models can be downloaded again.

Backup script, Linux and the Ubuntu shell on Windows. It leaves the big model cache out:

cat > ~/docker_volumes/vllm/backup.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
cd ~/docker_volumes/vllm
umask 077
tar czf "backups/vllm-config-$(date +%F).tar.gz" docker-compose.yml .env config
find backups -name 'vllm-config-*.tar.gz' -mtime +30 -delete
EOF
chmod +x ~/docker_volumes/vllm/backup.sh

The archive holds .env, so it is private (the umask 077 line makes it readable by you only). Do not upload it anywhere public.

Linux#

Schedule it at 03:00:

( crontab -l 2>/dev/null; echo "0 3 * * * $HOME/docker_volumes/vllm/backup.sh" ) | crontab -

Windows (PowerShell)#

Create a Task Scheduler task that runs the script inside WSL every day at 03:00:

$a = New-ScheduledTaskAction -Execute "wsl.exe" -Argument "-d Ubuntu -- bash -lc '~/docker_volumes/vllm/backup.sh'"
$t = New-ScheduledTaskTrigger -Daily -At 3am
Register-ScheduledTask -TaskName "vllm-backup" -Action $a -Trigger $t

Mac (Terminal)#

A Mac normally only holds a client key. Back up that key in your password manager. If you run the script on a server, use the cron line from Linux.

Restore test: unpack into a throwaway folder and check that Compose accepts it.

mkdir -p /tmp/vllm-restore && tar xzf ~/docker_volumes/vllm/backups/vllm-config-$(date +%F).tar.gz -C /tmp/vllm-restore
cd /tmp/vllm-restore && docker compose config > /dev/null && echo restore ok
rm -rf /tmp/vllm-restore

Performance and resources#

Tune against your own workload and change one thing at a time.

Measure with your own prompts, answer lengths and number of users. Then change one thing at a time: make the model fit first, set only the context you need, raise --max-num-seqs a little at a time, raise --gpu-memory-utilization only with free GPU memory, use prefix caching for repeated prompt starts, and use tensor parallelism only with several GPUs (on a tiny model it can be slower).

Monitor:

docker stats vllm
nvidia-smi
curl -s http://127.0.0.1:8000/metrics | head -40

/metrics is Prometheus format. Useful names include vllm:kv_cache_usage_perc (1 means 100%, not 1%), vllm:e2e_request_latency_seconds and vllm:inter_token_latency_seconds, plus queue depth. Keep /metrics private. To scrape it, point Prometheus at vllm:8000 on a shared Docker network.

FAQ#

Can I run this on an 8 GB NVIDIA GPU? Start with a small model like Qwen/Qwen3-0.6B. A 0.6B model fitting says nothing about a 7B model, so measure.

Do I need the CUDA Toolkit on the host? No, not for the prebuilt image. Install a current NVIDIA driver and the NVIDIA Container Toolkit. The image brings the rest. Building vLLM from source is a different job.

Is it the same as the OpenAI API? The request format is compatible. The answers come from your model, so they will differ from a hosted one.

Can I run the same Docker command on my Mac? Not with GPU speed. There is no CUDA on a Mac. Use the Mac as a client for a GPU machine, try vLLM-Metal (a community plugin for Apple Silicon), or use Ollama or llama.cpp.

Is it safe to open port 8000 to the internet if I set an API key? No. The key does not cover every route. Keep the port on 127.0.0.1, or publish only /v1/ through HTTPS and add rate limits.

Is vLLM faster than Ollama? It depends on the model, precision, context and number of users. We have no measured comparison, so test both on your own workload.

Sources#

• vLLM Docker deployment (page dated 2026-08-11, read on 2026-10-26)
• vLLM GPU installation and CPU installation (2026-10-08 and 2026-10-09)
• vLLM serve CLI, security, metrics, troubleshooting, optimization, quantization
• vLLM releases on GitHub (v0.32.0 of 2026-10-18) and the issue tracker
• vllm/vllm-openai on Docker Hub
• NVIDIA Container Toolkit install guide (undated page)
• Hugging Face environment variables

Checked against the official docs on 2026-10-26. Not run by DIY Lab Hub. The Windows setup steps, the Compose GPU reservation block and the Traefik labels come from general Docker, Microsoft and Traefik practice rather than the vLLM docs, so check them against your versions.