llama.cpp server in Docker: an OpenAI-compatible API on your GPU (Linux, Windows, Mac)

What you get in one minute#

At a glance:

• What you get: a private, OpenAI-compatible API plus a built-in web chat, served from one Docker container.
• Time needed: about 20 minutes, plus the image pull and the first model download.
• Difficulty: easy on Linux, medium on Windows (WSL2). A Mac runs it natively instead of in Docker.
• You need: Docker with Compose, and for GPU speed an NVIDIA card (or a Vulkan-capable GPU).
• You end up with: http://127.0.0.1:8080 for the chat page and http://127.0.0.1:8080/v1 for your apps.

llama.cpp is an open-source C/C++ engine that runs GGUF language models on your own hardware, and its llama-server program turns one into an OpenAI-compatible API with a web chat, in a single small container.

People pick it because it is light, it starts fast, and it runs almost anywhere: NVIDIA, Vulkan GPUs, plain CPUs and Apple Silicon. The -hf flag downloads a model from Hugging Face on the first start, so there is no manual download step. Your apps keep using the OpenAI SDK or curl. Only the base URL changes.

Why b11889 and not the newest tag? llama.cpp has no slow release cycle: every merged change becomes a new bNNNN tag. The Docker images are built after the release, so the very newest tag can be missing for a few hours. We checked that server, server-cuda, server-cuda13 and server-vulkan all exist for b11889. We also scanned the issue tracker for the last two weeks. The bug reports are tied to specific backends or models (SYCL, ROCm, Vulkan on one GPU, Metal with one model, a slowdown after one update). We found no wave of install failures or lost data. So the guide uses a recent release. If a later tag misbehaves for you, go back to an older tag. It is a one-line change.

How it fits together#

Your app sends an OpenAI-style request to the container, the container runs the model on your GPU, and the model file comes from a cache folder that Hugging Face fills on the first start.

Before you start#

You need Docker with Compose. For GPU speed on Linux or Windows you also need an NVIDIA card with a current driver, or another GPU that works with Vulkan.

There is no fixed VRAM minimum. Model size, context length and the number of parallel slots all use memory. The guide starts with a tiny model so it runs on almost anything.

Check the tools:

docker --version
docker compose version
nvidia-smi

• [ ] nvidia-smi shows your GPU (skip on Mac, or if you use CPU or Vulkan)
• [ ] Port 8080 is free
• [ ] Any old llama.cpp folder is backed up
• [ ] About 5 GB free disk for the image and a small model (more for bigger ones)
• [ ] Only for the Traefik section: a domain, DNS and ports 80/443

Tested on: not run by DIY Lab Hub for this guide (Linux = not run, Windows = not run, Mac = not run). Every command was checked against the official llama.cpp docs on 2026-10-26 (see Sources).

Choose your image#

For NVIDIA take server-cuda; for AMD or Intel GPUs on Linux take server-vulkan; with no GPU take server.

Add -b11889 to any name to pin a build, for example server-vulkan-b11889. The official docs say the GPU images are only built, not tested by CI, so a small first test is worth the two minutes.

Step 1: Make Docker see your GPU#

Docker must be able to use the GPU before llama-server can.

Linux#

NVIDIA only. These commands are for Ubuntu or Debian and follow NVIDIA's install guide:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

The restart briefly stops running containers. Other distributions use other package commands (see Sources).

For AMD or Intel with Vulkan you skip the toolkit. The compose file in Step 3 passes /dev/dri instead.

Test it:

docker run --rm --gpus all ubuntu nvidia-smi

Windows (PowerShell)#

1. Install a current NVIDIA driver on Windows. WSL2 GPU support comes with it. Do not install a Linux driver inside WSL.
2. Install Docker Desktop and keep "Use the WSL 2 based engine" on.
3. Switch to Linux containers.

Docker Desktop manages its own engine, so there is no nvidia-ctk step. Test it:

docker run --rm --gpus all ubuntu nvidia-smi

Mac (Terminal)#

Skip Docker for the GPU part. Install llama.cpp natively so it can use Metal:

brew install llama.cpp
llama-server --version

The official install page also lists MacPorts, Nix and conda-forge. Native llama-server takes the same flags as the container does, so Steps 3 to 5 still help you. You just run one command instead of Compose (shown in Step 3).

Check it worked: on Linux and Windows the test prints your GPU table. On Mac llama-server --version prints a version.

Potential problem: could not select device driver "" with capabilities: [[gpu]] means the toolkit is not configured. Repeat the nvidia-ctk line and restart Docker.

Step 2: Create folders and secrets#

Make the folders under docker_volumes/llamacpp and a .env file that holds your API key.

The key is what your apps send to your server. It protects the model routes. Never reuse another password.

Linux#

mkdir -p ~/docker_volumes/llamacpp/{data/models,config,logs,backups}
cd ~/docker_volumes/llamacpp
printf 'LLAMA_API_KEY=%s\n' "$(openssl rand -hex 32)" > .env
chmod 600 .env

Windows (PowerShell)#

$d = "$HOME\docker_volumes\llamacpp"
New-Item -ItemType Directory -Force "$d\data\models","$d\config","$d\logs","$d\backups" | Out-Null
Set-Location $d
$k = -join ((48..57 + 97..102) | Get-Random -Count 64 | ForEach-Object { [char]$_ })
"LLAMA_API_KEY=$k" | Set-Content -Encoding ascii .env

Mac (Terminal)#

Same commands as Linux:

mkdir -p ~/docker_volumes/llamacpp/{data/models,config,logs,backups}
cd ~/docker_volumes/llamacpp
printf 'LLAMA_API_KEY=%s\n' "$(openssl rand -hex 32)" > .env
chmod 600 .env

Check it worked: .env has one line starting with LLAMA_API_KEY=. Never commit it or paste it anywhere.

Step 3: Write the Compose file and start#

One service, one pinned image, the GPU reserved, the model cache on disk, and the port bound to loopback.

Linux#

Create docker-compose.yml in ~/docker_volumes/llamacpp. It has no version: line.

services:
  llamacpp:
    image: ghcr.io/ggml-org/llama.cpp:server-cuda-b11889
    container_name: llamacpp
    restart: unless-stopped
    env_file:
      - .env
    environment:
      LLAMA_CACHE: /models
    ports:
      - "127.0.0.1:8080:8080"
    volumes:
      - ./data/models:/models
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command:
      - -hf
      - Qwen/Qwen3-0.6B-GGUF:Q8_0
      - --alias
      - local-chat
      - --host
      - 0.0.0.0
      - --port
      - "8080"
      - -c
      - "4096"
      - -ngl
      - "99"
      - -np
      - "2"
      - --jinja

What each part does:

For Vulkan, change the image to server-vulkan-b11889, remove the deploy: block, and add:

    devices:
      - /dev/dri:/dev/dri

Start it and watch the log:

docker compose config
docker compose up -d
docker compose logs -f llamacpp

The first start downloads the model. Press Ctrl+C to leave the log. The container keeps running.

Windows (PowerShell)#

Use the same docker-compose.yml as the Linux step and save it in $HOME\docker_volumes\llamacpp. Then:

docker compose config
docker compose up -d
docker compose logs -f llamacpp

Docker Desktop passes the GPU through WSL2. If you prefer Linux paths, run the Linux commands inside Ubuntu (wsl -d Ubuntu) instead.

Mac (Terminal)#

Run it natively. No Compose, no container:

export LLAMA_API_KEY="$(grep LLAMA_API_KEY ~/docker_volumes/llamacpp/.env | cut -d= -f2)"
llama-server -hf Qwen/Qwen3-0.6B-GGUF:Q8_0 --alias local-chat \
  --host 127.0.0.1 --port 8080 -c 4096 -ngl 99 -np 2

On Apple Silicon the Metal backend is used automatically. Leave the terminal open, or ask launchd to start it for you.

Check it worked: the log shows the server listening, and this prints {"status":"ok"} (the health route is public):

curl -s http://127.0.0.1:8080/health

While the model loads you get HTTP 503 with the message "Loading model". That is normal. Wait and ask again.

Step 4: Open the web chat and test the API#

Open the page in a browser, then send one request with your key.

1. Open http://127.0.0.1:8080 in your browser. This is the built-in web UI (on by default).
2. If it asks for a key, paste the value from your .env file.
3. Send a message.

Linux#

set -a; . ~/docker_volumes/llamacpp/.env; set +a
curl -s http://127.0.0.1:8080/v1/models -H "Authorization: Bearer $LLAMA_API_KEY"
curl -s http://127.0.0.1:8080/v1/chat/completions \
  -H "Authorization: Bearer $LLAMA_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"local-chat","messages":[{"role":"user","content":"Explain what an inference server does in two sentences."}],"max_tokens":256}'

Windows (PowerShell)#

$key = (Get-Content "$HOME\docker_volumes\llamacpp\.env" | Select-String '^LLAMA_API_KEY=').Line.Split('=')[1]
$h = @{ Authorization = "Bearer $key" }
Invoke-RestMethod http://127.0.0.1:8080/v1/models -Headers $h | ConvertTo-Json -Depth 5
$body = @{ model = "local-chat"; max_tokens = 256; messages = @(@{ role = "user"; content = "Explain what an inference server does in two sentences." }) } | ConvertTo-Json -Depth 6
Invoke-RestMethod -Method Post http://127.0.0.1:8080/v1/chat/completions -Headers $h -ContentType "application/json" -Body $body | ConvertTo-Json -Depth 8

The real curl is curl.exe. Plain curl in PowerShell is an alias.

Mac (Terminal)#

Same commands as Linux, with the LLAMA_API_KEY variable from Step 3.

Check it worked: the first call lists local-chat. The second returns the usual OpenAI-style JSON with the answer in choices[0].message.content. Without the key you get a 401 error. Qwen3 may write its thinking before the answer. That is normal for that model.

In any OpenAI SDK set the base URL to http://127.0.0.1:8080/v1, the key to your LLAMA_API_KEY and the model to local-chat. For one address in front of several backends, see our LiteLLM guide. If you need heavy multi-user throughput on NVIDIA, compare our vLLM guide.

The server also speaks more than chat. The official README lists /v1/completions, /v1/responses, a text-numbers endpoint at /v1/embeddings (needs a model started for that job) and an Anthropic-style /v1/messages.

llama.cpp or Ollama?#

Ollama is the easier daily driver, and llama.cpp gives you more control and the newest features first. Ollama is built on llama.cpp. Our Ollama guide covers it.

Put it behind HTTPS (with or without Traefik)#

With Traefik#

On a server with a domain, let Traefik publish the API over HTTPS and remove the host port.

This is for a machine with a domain. Local-only users can skip it. It assumes Traefik v3 with the Docker provider, an entrypoint called https and a certificate for your hostname. Create the network once if it is missing:

docker network create proxy

In docker-compose.yml delete the ports: block and merge in:

services:
  llamacpp:
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.llamacpp.rule=Host(`llm.<your-domain>`) && PathPrefix(`/v1/`)"
      - "traefik.http.routers.llamacpp.entrypoints=https"
      - "traefik.http.routers.llamacpp.tls=true"
      - "traefik.http.services.llamacpp.loadbalancer.server.port=8080"
networks:
  proxy:
    external: true

Add traefik.http.routers.llamacpp.tls.certresolver=<your-resolver> if you use ACME. The PathPrefix rule is on purpose: /health is public by design, and the web chat page is not behind the key, so only /v1/ goes out.

Check it worked: curl -I https://llm.<your-domain>/v1/models answers 401 without a key.

Without Traefik: Caddy#

Choose Caddy when you have no Traefik yet and run one small server. Facts are from caddyserver.com/docs.

Public domain, Caddyfile for the llama.cpp server:

app.example.com {
    reverse_proxy /v1/* localhost:8080
}

Home only:

app.home.arpa {
    tls internal
    reverse_proxy /v1/* localhost:8080
}

Trust the root certificate (home only): on the Caddy machine run sudo caddy trust (Linux). On Windows and Mac copy pki/authorities/local/root.crt from Caddy's data folder and import it as a trusted root (Windows certificate manager, Mac Keychain Access).

Run Caddy (2.11.7 was current on 2026-10-26): Linux with the official apt repository from the install page, then sudo systemctl reload caddy; Windows winget install CaddyServer.Caddy then caddy run; Mac brew install caddy then caddy run. Or use Docker: put caddy:2.11.7 on the same network as the app, drop the app's ports: line and write reverse_proxy llamacpp:8080.

services:
  caddy:
    image: caddy:2.11.7
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ~/docker_volumes/caddy/data:/data
      - ~/docker_volumes/caddy/config:/config

WebSockets work without extra settings. Only /v1/* is passed on, like the Traefik rule above, so /health and the web chat page stay private.

Check it worked: curl -I https://app.example.com answers HTTP/2 200 (or the login redirect). On Windows use curl.exe.

Security and testing#

The API key protects the model routes, so keep the port private anyway.

• Keep .env at chmod 600 and out of git.
• Bind to 127.0.0.1, or publish only /v1/ through Traefik. Do not forward port 8080 on your router.
• Leave the experimental built-in tool and proxy options off on a shared machine. The official README says "do not enable in untrusted environments" for them.
• The /metrics and /props routes are off unless you add --metrics or --props. Leave them off unless you need them.
• Pin the image tag. Do not run latest or an unpinned server-cuda on a server people use.
• Huge requests can use up your VRAM, so rate limits belong at the gateway.

Quick tests after every change:

docker compose ps
docker compose logs --tail=100 llamacpp
curl -s http://127.0.0.1:8080/health

Troubleshooting#

Read the first error in docker compose logs llamacpp before you change anything.

Known bugs#

The issue tracker in the two weeks before this guide shows reports that depend on the setup: a SYCL (Intel Arc) NaN-output bug, a ROCm compile error, a crash with one option on Vulkan under Windows, a Metal crash with one large model, and a slowdown after one update. If you use one of those backends, test first. The tracker is in Sources.

Useful diagnostics#

docker logs llamacpp --tail 100
docker compose ps
docker inspect llamacpp --format '{{.State.Status}}'

Updates and backups#

To update, change the tag in the compose file, pull and restart. Back up the folder, not the container.

1. Pick the new tag from the releases page. Check that the image exists on the package page first.
2. Edit the image line.
3. Run:

cd ~/docker_volumes/llamacpp
docker compose pull
docker compose up -d

On Windows use the same three commands in PowerShell, after Set-Location $HOME\docker_volumes\llamacpp.

Backup script#

Models can be downloaded again, so the important parts are docker-compose.yml and .env. The script below skips the large model files:

#!/usr/bin/env bash
set -euo pipefail
cd ~/docker_volumes/llamacpp
tar --exclude='data/models' -czf backups/llamacpp-$(date +%F).tar.gz docker-compose.yml .env config
find backups -name 'llamacpp-*.tar.gz' -mtime +14 -delete

Linux and Mac: save it as backup.sh, make it executable, and add it to cron, for example 0 3 * * * /home/<you>/docker_volumes/llamacpp/backup.sh. Windows: run the same tar line from Task Scheduler at 03:00 (tar ships with Windows 11).

Restore test#

Test the backup in a throwaway folder, never over your live one:

mkdir -p /tmp/restore-test
tar -xzf ~/docker_volumes/llamacpp/backups/llamacpp-$(date +%F).tar.gz -C /tmp/restore-test
ls -a /tmp/restore-test
rm -rf /tmp/restore-test

You should see docker-compose.yml, .env and config. Treat the unpacked .env as a secret.

Performance and resources#

GPU offload decides most of your speed; context size and slots decide your memory.

Watch it live:

docker stats llamacpp
nvidia-smi

DIY Lab Hub has not measured tokens per second for this guide, so we give no speed numbers. Measure your own model on your own card.

FAQ#

Is llama-server really OpenAI-compatible? For chat, completions and the model list, yes. Point the OpenAI SDK at /v1. A local model can still answer differently from a hosted one, so test your app with the model you choose.

Can I run several models? One server runs one model by default. The official README also describes a "router" mode with --models-dir, and you can simply start more containers on other ports.

Why does a Mac container not use the GPU? Docker on a Mac has no Metal access. Install llama.cpp natively with Homebrew and it uses Metal automatically.

Can I use my own GGUF file instead of -hf? Yes. Put the file in data/models and replace the -hf lines with -m /models/<file>.gguf.

Do I need the web UI? No. Add --no-webui to turn it off and keep only the API.

Sources#

Read on 2026-10-26:

• llama.cpp Docker docs (image names and Docker usage)
• llama-server README (flags, environment variables, endpoints, compose example)
• Install pre-built llama.cpp (Homebrew, Winget, conda-forge)
• llama.cpp releases and container packages (tag b11889 and image tags checked through the GitHub and ghcr.io APIs)
• llama.cpp issue tracker (breakage check)
• NVIDIA Container Toolkit install guide
• Qwen3-0.6B-GGUF on Hugging Face (file Qwen3-0.6B-Q8_0.gguf, not gated)
• Caddy docs: automatic HTTPS, tls internal, reverse_proxy, install (read on 2026-10-26)

Checked against the official docs on 2026-10-26. Not run by DIY Lab Hub.