SGLang with Docker: a fast OpenAI-compatible LLM server on your NVIDIA GPU (Linux, Windows, Mac)

What you get in one minute#

SGLang is an open-source LLM serving engine that loads a model from Hugging Face and answers OpenAI-style requests on your own NVIDIA GPU; the official Docker image makes it a one-container job.

Logo: the official SGLang project (github.com/sgl-project/sglang)

Your apps keep using the OpenAI SDK or plain curl; only the base URL changes to http://127.0.0.1:30000/v1. The project is active, not archived or renamed: LMSYS, a non-profit, hosts it and it shipped a release on 2 November 2026. Only the docs moved: docs.sglang.ai now forwards to docs.sglang.io.

If you know our vLLM guide, this is the same idea with a different engine. Neither project's docs say which is faster on your card, so we make no speed claim.

At a glance:

• What it is: an inference server with no chat page; it is the engine behind an app.
• What you need: Linux, an NVIDIA GPU, a driver of version 580 or newer, Docker and the NVIDIA Container Toolkit.
• API: OpenAI-compatible; the docs also list Anthropic- and Ollama-compatible endpoints.
• Windows: no Windows steps in the docs. Use it as a client (WSL2 is at your own risk).
• Mac: no Docker GPU path; the docs have a separate Apple Silicon route (MLX), covered below.
• Privacy: after the one-time model download, requests stay on your machine.

How it fits together#

Your app sends an OpenAI-style request to the SGLang container, which runs the model on your GPU. The model files come from a cache folder that Hugging Face fills on the first start.

Drawn for this guide from the official quickstart and install pages. The dashed HTTPS box is optional.

Before you start#

You need an NVIDIA GPU, a driver of version 580 or newer, and Docker with the NVIDIA Container Toolkit. The official image is built on CUDA 13 and CUDA 12 images are no longer published.

Which GPUs? The README names examples from B200 to RTX 30, 40 and 50 series and A100, and the install page says FlashInfer, the default attention backend, supports compute capability 7.5 and above. There is no VRAM minimum. We start with Qwen/Qwen3-0.6B, the quickstart's small model: 751,632,384 BF16 parameters, roughly 1.5 GB of weights (our arithmetic). The image is the big part, about 16 GB.

Check the tools on the Linux server:

docker --version
docker compose version
nvidia-smi

• [ ] nvidia-smi shows your GPU and a driver version of 580 or newer
• [ ] Docker can reach the GPU (Step 1)
• [ ] Port 30000 is free on the machine
• [ ] About 20 GB or more of free disk space
• [ ] For the HTTPS section only: a domain, a DNS record and ports 80 and 443

Tested by DIY Lab Hub: not yet (Linux = not run, Windows = not run, Mac = not run). Commands come from the official SGLang, NVIDIA and Caddy docs and the v0.5.23 repository files, read on 5 November 2026.

Step 1: Make Docker see your GPU#

Docker must be able to run nvidia-smi inside a container before SGLang has any chance.

If nvidia-smi already fails on the host, stop and fix the driver first. The container toolkit cannot repair a broken driver.

Linux#

Install the NVIDIA Container Toolkit and configure the Docker runtime. We already wrote this step out in full in Step 1 of our vLLM guide (Ubuntu and Debian commands from NVIDIA's install guide); the official SGLang install page links to the same NVIDIA guide. Then pull the pinned image and test it:

docker pull lmsysorg/sglang:v0.5.23
docker run --rm --gpus all lmsysorg/sglang:v0.5.23 nvidia-smi

Windows (PowerShell)#

Not supported by the SGLang docs, which describe Linux and have no Windows or WSL2 page. You could try the Linux commands inside an Ubuntu WSL2 shell with Docker Desktop; we did not test that, and issue 41882 (open) reports a WSL2 crash with two GPUs. Better: run the server on Linux and use Windows as the client in Step 4.

docker --version

Mac (Terminal)#

Not supported with Docker: a Mac has no NVIDIA GPU or CUDA. Use a Linux server, or see the Apple Silicon section below.

Check it worked. On Linux, the second command prints the usual nvidia-smi table with your GPU. The 16 GB pull can take a long time on a slow link.

Potential problem: the Docker error could not select device driver "" with capabilities: [[gpu]] means the toolkit runtime is not configured; repeat the nvidia-ctk runtime configure line from NVIDIA's guide and restart Docker.

Tip: Use a pinned tag like v0.5.23 and not latest. Today they are the same image, but latest moves with every release, and you want to decide when your server changes.

Step 2: Create the folders and the API key#

Make the folders under ~/docker_volumes/sglang and a .env file with a random API key.

The API key is what your own apps send to your server. It is not a Hugging Face token.

Linux#

mkdir -p ~/docker_volumes/sglang/{data/hf-cache,config,logs,backups}
cd ~/docker_volumes/sglang
printf 'SGLANG_API_KEY=%s\n' "$(openssl rand -hex 32)" > .env
chmod 600 .env

Windows (PowerShell)#

Not supported as the server (Step 1). If you tried WSL2 anyway, run the Linux commands in the Ubuntu shell. As a client you only need the key from the server's owner: keep it in a variable for the session, never in a script you share.

$env:SGLANG_API_KEY = Read-Host "SGLang API key"

Mac (Terminal)#

As a client you only need the key to call a remote server:

read -rsp "SGLang API key: " SGLANG_API_KEY; export SGLANG_API_KEY; echo

Check it worked. On the server, ls -l ~/docker_volumes/sglang/.env shows -rw------- and one line SGLANG_API_KEY=.... Never commit the file or paste it into a chat.

Tip: Give apps only the API key. A Hugging Face token belongs to your account and is only for downloading gated models.

Step 3: Write the Compose file and start SGLang#

One service, one pinned image, the GPU reserved, the model cache on disk and the port bound to loopback.

This follows the official docker/compose.yaml at the v0.5.23 tag. That file uses host networking and privileged: true, marked "required by RDMA", and offers a plain ports: mapping, which we use. We pin the image, add an API key, bind to 127.0.0.1 and keep the official /health check (the timings are ours). We mount only the Hugging Face cache folder, as the install page does.

Linux#

Create docker-compose.yml in ~/docker_volumes/sglang. It has no version: line.

services:
  sglang:
    image: lmsysorg/sglang:v0.5.23
    container_name: sglang
    restart: unless-stopped
    ipc: host
    ports:
      - "127.0.0.1:30000:30000"
    volumes:
      - ./data/hf-cache:/root/.cache/huggingface
    command:
      - sglang
      - serve
      - Qwen/Qwen3-0.6B
      - --host
      - 0.0.0.0
      - --port
      - "30000"
      - --api-key
      - ${SGLANG_API_KEY:?set SGLANG_API_KEY in .env}
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost:30000/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 5
      start_period: 300s
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: 1
              capabilities: [gpu]

Start it and watch the log:

cd ~/docker_volumes/sglang
docker compose up -d
docker compose logs -f sglang

Why: --host 0.0.0.0 is needed inside the container because the default host is 127.0.0.1. The 127.0.0.1: in ports: keeps the port off your network. ipc: host gives the shared memory the docs ask for. Qwen/Qwen3-0.6B is not gated (Apache-2.0).

Windows (PowerShell)#

Not supported as the server (Step 1). Use the Linux file on a Linux machine.

Mac (Terminal)#

Not supported with Docker. See Step 1.

Check it worked. The docs say to wait for the log line The server is fired up and ready to roll! (expected text from the official quickstart, not our own run). The first start downloads the model into data/hf-cache, so it takes longer. docker compose ps then shows sglang running, and healthy once the health check passes.

Tip: Ctrl+C leaves docker compose logs -f; the server keeps running.

Step 4: Test the API#

List the models, then send one chat request with your key.

Linux#

set -a; . ~/docker_volumes/sglang/.env; set +a
curl -s http://127.0.0.1:30000/v1/models -H "Authorization: Bearer $SGLANG_API_KEY"
curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H "Authorization: Bearer $SGLANG_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"What is the capital of France?"}]}'

Windows (PowerShell)#

This is the normal Windows role: a client for a server somewhere else. Replace the address with your server's (use the HTTPS name once you have it). If the server runs on this PC in WSL2, 127.0.0.1:30000 normally works too.

$h = @{ Authorization = "Bearer $env:SGLANG_API_KEY" }
$base = "http://<server-address>:30000"
Invoke-RestMethod "$base/v1/models" -Headers $h | ConvertTo-Json -Depth 5
$body = @{ model = "Qwen/Qwen3-0.6B"; messages = @(@{ role = "user"; content = "What is the capital of France?" }) } | ConvertTo-Json -Depth 6
Invoke-RestMethod -Method Post "$base/v1/chat/completions" -Headers $h -ContentType "application/json" -Body $body | ConvertTo-Json -Depth 8

Plain curl in PowerShell is an alias; call curl.exe if you want the real one.

Mac (Terminal)#

Same commands as Linux. For a remote server replace 127.0.0.1 with its address and prefer the HTTPS name:

curl -s http://127.0.0.1:30000/v1/models -H "Authorization: Bearer $SGLANG_API_KEY"
curl -s http://127.0.0.1:30000/v1/chat/completions \
  -H "Authorization: Bearer $SGLANG_API_KEY" -H "Content-Type: application/json" \
  -d '{"model":"Qwen/Qwen3-0.6B","messages":[{"role":"user","content":"What is the capital of France?"}]}'

Check it worked. The second call returns OpenAI-style JSON; the docs say the answer is in choices[0].message.content. Without the key you get an HTTP 401 error. Qwen3 models can write their reasoning before the answer; the official install page shows --reasoning-parser qwen3 for this model if you want it separated.

Any OpenAI SDK works: base URL http://127.0.0.1:30000/v1, your key, model Qwen/Qwen3-0.6B. "OpenAI-compatible" does not mean a small local model answers like a hosted one. For one address over several backends see our LiteLLM guide; for a coding assistant on a local model see Aider or Cline.

Tip: Use the exact model name you started the server with. The docs' own example sends the model path as the model field.

If you have an Apple Silicon Mac#

A Mac cannot use the Docker image, but SGLang's docs describe running it natively on Apple Silicon through MLX. We did not run this route; it is the docs' own route, shortened.

The docs list macOS 14 or newer and a from-source install. The optional native Metal kernels need the full Xcode app.

git clone https://github.com/sgl-project/sglang.git
cd sglang
uv venv -p 3.12 sglang-metal
source sglang-metal/bin/activate
rm -f python/pyproject.toml && mv python/pyproject_other.toml python/pyproject.toml
uv pip install -e "python[all_mps]"
SGLANG_USE_MLX=1 python -m sglang.launch_server \
  --model <MODEL_ID_OR_PATH> \
  --disable-cuda-graph \
  --host 127.0.0.1

The docs use --host 0.0.0.0 here; we changed it to 127.0.0.1 so the server is not offered to your whole network. Read the docs page first. For a simple local model on a Mac, Ollama is easier.

Put it behind HTTPS (with or without Traefik)#

Use this only if other devices must reach the server. Put HTTPS in front and publish only /v1/.

Local-only users can skip this section and use http://127.0.0.1:30000. Both routes need a domain with a DNS record pointing to the server and ports 80 and 443 reachable. The routes below publish only the /v1/ API. We read in the source at v0.5.23 that paths starting with /health, /ready and /metrics are allowed without an API key, so keep those private.

With Traefik#

On a server that already runs Traefik v3 with the Docker provider, add labels and drop the host port.

This assumes an external Docker network called proxy and an entrypoint called https (use your own names). Create the network once:

docker network create proxy

Change the service in ~/docker_volumes/sglang/docker-compose.yml: delete the ports: block, then add the network and labels. Keep everything else as in Step 3.

services:
  sglang:
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.sglang.rule=Host(`llm.<your-domain>`) && PathPrefix(`/v1/`)"
      - "traefik.http.routers.sglang.entrypoints=https"
      - "traefik.http.routers.sglang.tls=true"
      - "traefik.http.services.sglang.loadbalancer.server.port=30000"

networks:
  proxy:
    external: true

Add traefik.http.routers.sglang.tls.certresolver=<your-resolver> if you use ACME. HTTPS adds no rate limiting. Apply with docker compose up -d.

Check it worked. curl -I https://llm.<your-domain>/v1/models answers 401 without a key, and the same request with -H "Authorization: Bearer $SGLANG_API_KEY" answers 200.

Without Traefik: Caddy#

Choose Caddy if you have no Traefik. It is one small file and it gets the certificate for you.

For a public domain, point a DNS A record at the server and open ports 80 and 443. For a home-only name, tls internal makes a local certificate that each device must trust (below).

Create a Caddy folder. Caddy's Docker docs say to bind-mount the conf folder that holds the Caddyfile, not the single file:

mkdir -p ~/docker_volumes/caddy/{conf,data,config}
cd ~/docker_volumes/caddy

Save this as ~/docker_volumes/caddy/docker-compose.yml. caddy:2.11.8 is the latest Caddy release we saw on 5 November 2026:

services:
  caddy:
    image: caddy:2.11.8
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80/tcp"
      - "443:443/tcp"
      - "443:443/udp"
    volumes:
      - ./conf:/etc/caddy
      - ./data:/data
      - ./config:/config
    networks:
      - proxy

networks:
  proxy:
    external: true

Save this as ~/docker_volumes/caddy/conf/Caddyfile for a public domain:

llm.example.com {
    handle /v1/* {
        reverse_proxy sglang:30000
    }
    handle {
        respond 404
    }
}

For a home-only name (use a name your own DNS or hosts file points at this machine; .home.arpa is reserved for home networks), use tls internal:

llm.home.arpa {
    tls internal
    handle /v1/* {
        reverse_proxy sglang:30000
    }
    handle {
        respond 404
    }
}

Caddy reaches SGLang by container name on the shared proxy network, so SGLang needs no host port. In ~/docker_volumes/sglang/docker-compose.yml, delete the ports: block and add networks: [proxy] plus the proxy network entry, exactly as in the Traefik block but without the labels. Then start both:

cd ~/docker_volumes/caddy && docker compose up -d
cd ~/docker_volumes/sglang && docker compose up -d

For a home-only certificate, find Caddy's root certificate and copy it out, then install it as a trusted root on each device that will connect:

docker exec caddy find /data -name root.crt
docker cp caddy:<path-from-find> ./caddy-root.crt

Install it as a trusted root: on Windows into "Trusted Root Certification Authorities", on a Mac in Keychain Access with "Always Trust", on Linux in your distribution's CA store (common practice, not from the SGLang docs). Run the docker and Caddy commands on the Linux server only.

Check it worked. curl -I https://llm.example.com/v1/models returns an HTTP status line over HTTPS (401 without a key). curl -s -o /dev/null -w '%{http_code}\n' https://llm.example.com/metrics returns 404 because only /v1/ is published. For a home-only name the browser or curl accepts the certificate only after the root is trusted on that device.

Tip: A certificate error usually means the name in the Caddyfile differs from the DNS record, or ports 80 and 443 do not reach the server.

Security and testing#

An inference server with no key is an open GPU. These controls matter most.

Quick tests on Linux after each change:

cd ~/docker_volumes/sglang
docker compose ps
docker compose logs --tail=100 sglang
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:30000/v1/models
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:30000/v1/models -H "Authorization: Bearer $SGLANG_API_KEY"

The first status should be 401 and the second 200. Windows and Mac can run the same curl check against your HTTPS name (curl.exe on Windows).

What leaves the machine: the image pull and the model download, once. The docs describe no telemetry; we did not inspect traffic.

Common mistakes#

These come from the official docs' own warnings, SGLang's issue tracker (read 5 November 2026) and common AI assistant answers.

Tip: Run docker run --rm --gpus all lmsysorg/sglang:v0.5.23 nvidia-smi before anything else. If that fails, no SGLang setting will help.

Tip: If the server seems stuck, read docker compose logs --tail=100 sglang first. The first start downloads a model, so a quiet log can be normal.

Tip: Do not paste a recipe's flag list unread. Many flags (parallelism, speculative decoding) are for large multi-GPU machines and can stop a small card from starting.

Troubleshooting#

Known bugs (open, November 2026)#

• Issue 39216: a client disconnecting during a very long request crashed the engine in one report (a development image, a huge model). A fix, pull request 41472, is linked in the thread. Match client timeouts to your longest prompts.
• Issues 41882, 38291, 45206: see the mistakes table.

Updates and backups#

Update. Read the release notes, change the tag in docker-compose.yml, then:

cd ~/docker_volumes/sglang
docker compose pull
docker compose up -d
docker compose logs --tail=100 sglang

The project ships often, so test a new tag with your own prompts before you rely on it.

Back up. The model files can be downloaded again, so skip data/hf-cache. Back up the compose file and .env (and the Caddy conf folder if you use it). Create ~/docker_volumes/sglang/backups/backup.sh:

#!/usr/bin/env bash
set -euo pipefail
cd ~/docker_volumes/sglang
mkdir -p backups
tar -czf "backups/sglang-$(date +%F).tar.gz" docker-compose.yml .env config
chmod 600 backups/sglang-*.tar.gz

If you use Caddy, back up ~/docker_volumes/caddy/conf the same way. Make it runnable with chmod +x ~/docker_volumes/sglang/backups/backup.sh. To run it nightly at 03:00 on Linux, add a crontab line with crontab -e:

0 3 * * * /home/<your-user>/docker_volumes/sglang/backups/backup.sh

Restore test. Unpack one archive into a new empty folder, never over the live one, and check the files:

mkdir -p /tmp/restore-test && tar -xzf ~/docker_volumes/sglang/backups/sglang-<date>.tar.gz -C /tmp/restore-test
ls -a /tmp/restore-test

Windows (PowerShell)#

Not for the server. Copy the archive off the Linux server from your PC, and use Task Scheduler for that copy if you want it nightly:

scp <your-user>@<your-server-address>:~/docker_volumes/sglang/backups/sglang-<date>.tar.gz .

Mac (Terminal)#

Use the same scp command from Terminal. On a Mac, schedule it with cron or launchd.

Performance and resources#

The docs publish no VRAM, CPU or speed figures for a small server, so this guide gives none. Measure your own:

• Live use: nvidia-smi shows GPU memory; docker stats sglang shows container CPU and RAM.
• Memory split: the tuning page says memory is weights, KV cache pool, CUDA graph buffers and activations. --mem-fraction-static sets the share for the first two; if unset, SGLang computes it (0.88 when it cannot read GPU memory).
• How to tune: reserve about 5 to 8 GB for activations and check available_gpu_mem in the log just before the server is ready: too high means you can raise the fraction, too low risks out-of-memory errors.
• Metrics: --enable-metrics serves Prometheus metrics at /metrics. Keep that path private.

SGLang or vLLM?#

Both give an OpenAI-compatible API on your NVIDIA GPU. The vLLM facts below come from our own vLLM guide (checked 26 October 2026), the SGLang facts from the official docs.

Pick the one whose supported models match what you want to run; try both if speed matters.

FAQ#

Is SGLang still maintained, or did it move? It is maintained: v0.5.23 came out on 2 November 2026. Only the docs moved, to docs.sglang.io.

Is it free to use? Yes, under Apache-2.0. Models have their own licences: check each model's page.

How much GPU memory do I need? The docs give no minimum. It depends on the model, precision, context length and parallel requests. The 0.6-billion-parameter model here needs roughly 1.5 GB for weights (our arithmetic) plus cache and overhead.

Can I run it on Windows or a Mac? Not with Docker. Windows works as a client of a Linux server (WSL2 untested by us). On Apple Silicon the docs describe a native MLX route that we did not run.

Does it work with the OpenAI SDK or other tools? Yes: anything that speaks an OpenAI-style /v1 API with a Bearer key. Set the base URL to your server.

Sources#

• SGLang README, release v0.5.23, LICENSE, Dockerfile, compose.yaml and auth.py - read 5 November 2026
• SGLang docs: Installation, Quickstart, Server arguments, Hyperparameter tuning, FAQ, Apple Silicon with Metal - read 5 November 2026
• Docker Hub: lmsysorg/sglang tags (v0.5.23 and latest checked) - 5 November 2026
• NVIDIA CUDA Toolkit release notes (minimum driver for CUDA 13) and NVIDIA Container Toolkit install guide - read 5 November 2026
• LMSYS: SGLang security update: improving request isolation - read 5 November 2026
• Caddy running in Docker, Caddy TLS directive and Docker Hub caddy image (tag 2.11.8) - read 5 November 2026
• Qwen/Qwen3-0.6B on Hugging Face (licence and parameter count) - read 5 November 2026
• SGLang issues #30856, #36058, #38291, #39216, #41882, #42917 and #45206 - read 5 November 2026

Checked against the official docs on 2026-11-05.