LiteLLM Proxy with Docker: one OpenAI-compatible API for Ollama and cloud models (Linux, Windows, Mac)

What you get in one minute#

LiteLLM Proxy is an open-source gateway that gives all your models, local Ollama and cloud providers alike, one OpenAI-compatible address on port 4000, so your apps switch models by changing a name.

Your apps talk to one URL with one kind of key. Behind it you list the models in a config.yaml. With Postgres added, you can create virtual keys per app or person, set budgets, see spend and use the Admin UI. The master key stays with you and the apps only get virtual keys.

How it fits together#

Your apps send an OpenAI-style request with a key to LiteLLM, LiteLLM checks the key and budget in Postgres and forwards the call to Ollama on your machine or to a cloud provider.

The second picture shows what a virtual key does once you create one.

Before you start#

You need Docker with Compose, and Ollama installed on the same machine if you want local models.

Check the tools:

docker --version
docker compose version

• [ ] Ollama runs and has a model: ollama pull llama3.1
• [ ] Port 4000 is free on the machine
• [ ] Anything you keep in an old LiteLLM folder is backed up
• [ ] Only for the Traefik section: a domain, DNS and ports 80/443

Tested on: not run by DIY Lab Hub for this guide (Linux = not run, Windows = not run, Mac = not run). The commands were checked against the official LiteLLM and Ollama docs on 2026-10-26 (see Sources).

Step 1: Create the folders and the secrets#

Make the folders under ~/docker_volumes/litellm and a .env file with three random secrets.

You need three secrets. LITELLM_MASTER_KEY is the root admin key (it must start with sk-). LITELLM_SALT_KEY encrypts provider keys that LiteLLM stores in the database; it cannot be rotated, and if it changes the stored credentials become unreadable. POSTGRES_PASSWORD protects the database. Hex strings are safe inside the database URL.

Linux#

mkdir -p ~/docker_volumes/litellm/{config,data,backups}
cd ~/docker_volumes/litellm
printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\nPOSTGRES_PASSWORD=%s\n' \
  "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" "$(openssl rand -hex 24)" > .env
chmod 600 .env

Windows (PowerShell)#

Use PowerShell 7. This generator draws real random bytes:

New-Item -ItemType Directory -Force "$HOME\docker_volumes\litellm\config","$HOME\docker_volumes\litellm\backups" | Out-Null
Set-Location "$HOME\docker_volumes\litellm"
function New-Hex($n) { -join ([System.Security.Cryptography.RandomNumberGenerator]::GetBytes($n) | ForEach-Object { $_.ToString('x2') }) }
@(
  "LITELLM_MASTER_KEY=sk-$(New-Hex 32)"
  "LITELLM_SALT_KEY=sk-$(New-Hex 32)"
  "POSTGRES_PASSWORD=$(New-Hex 24)"
) | Set-Content .env

On Windows the database lives in a named Docker volume (see Step 2), so there is no data folder to create.

Mac (Terminal)#

Same commands as Linux:

mkdir -p ~/docker_volumes/litellm/{config,data,backups}
cd ~/docker_volumes/litellm
printf 'LITELLM_MASTER_KEY=sk-%s\nLITELLM_SALT_KEY=sk-%s\nPOSTGRES_PASSWORD=%s\n' \
  "$(openssl rand -hex 32)" "$(openssl rand -hex 32)" "$(openssl rand -hex 24)" > .env
chmod 600 .env

If you also want a cloud model, add its key as one more line in .env, for example OPENAI_API_KEY=<YOUR_OPENAI_KEY>.

Check it worked: .env has three lines and (Linux/Mac) ls -l .env shows -rw-------. Never commit this file or paste its content anywhere.

Step 2: Write the config and the Compose file#

Put the model list in config/config.yaml and the two containers in docker-compose.yml.

In config.yaml, model_name is what your apps ask for. ollama_chat/ is the recommended prefix for Ollama (it uses the /api/chat endpoint). The value os.environ/NAME reads an environment variable, so no key is written into the file. Never use localhost in api_base: inside the container that is the container itself.

Create config/config.yaml with this content (all three systems):

model_list:
  - model_name: local-llama
    litellm_params:
      model: ollama_chat/llama3.1
      api_base: http://host.docker.internal:11434
  - model_name: gpt-cloud
    litellm_params:
      model: openai/gpt-4o-mini   # any model your account has
      api_key: os.environ/OPENAI_API_KEY
general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY

If you have no cloud key, delete the gpt-cloud entry. Now the Compose file. There is no version: line, the image tags are pinned and the published port stays on 127.0.0.1.

Linux#

cat > docker-compose.yml << 'EOF'
services:
  litellm:
    image: ghcr.io/berriai/litellm:v1.106.3
    restart: unless-stopped
    command: ["--config", "/app/config.yaml"]
    env_file:
      - .env
    environment:
      DATABASE_URL: postgresql://llmproxy:${POSTGRES_PASSWORD}@db:5432/litellm
    volumes:
      - ./config/config.yaml:/app/config.yaml:ro
    ports:
      - "127.0.0.1:4000:4000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    depends_on:
      db:
        condition: service_healthy
  db:
    image: postgres:17
    restart: unless-stopped
    environment:
      POSTGRES_USER: llmproxy
      POSTGRES_DB: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - ./data/postgres:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d litellm -U llmproxy"]
      interval: 5s
      timeout: 5s
      retries: 10
EOF

Save the YAML above as config/config.yaml the same way (nano config/config.yaml or a heredoc).

Windows (PowerShell)#

Keep the database in a named volume, because bind-mounting a Windows folder into Postgres through WSL-backed Docker often fails on permissions. Save the model list from above as config\config.yaml (any editor), then:

@'
services:
  litellm:
    image: ghcr.io/berriai/litellm:v1.106.3
    restart: unless-stopped
    command: ["--config", "/app/config.yaml"]
    env_file:
      - .env
    environment:
      DATABASE_URL: postgresql://llmproxy:${POSTGRES_PASSWORD}@db:5432/litellm
    volumes:
      - ./config/config.yaml:/app/config.yaml:ro
    ports:
      - "127.0.0.1:4000:4000"
    extra_hosts:
      - "host.docker.internal:host-gateway"
    depends_on:
      db:
        condition: service_healthy
  db:
    image: postgres:17
    restart: unless-stopped
    environment:
      POSTGRES_USER: llmproxy
      POSTGRES_DB: litellm
      POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
    volumes:
      - litellm_pgdata:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -d litellm -U llmproxy"]
      interval: 5s
      timeout: 5s
      retries: 10
volumes:
  litellm_pgdata:
'@ | Set-Content docker-compose.yml

Mac (Terminal)#

Same commands as Linux (the docker-compose.yml and config/config.yaml blocks are identical).

Docker Desktop on Windows and Mac already knows host.docker.internal; the extra_hosts line is harmless there and is what makes it work on Linux.

Check it worked: docker compose config prints the merged file without errors and without a version warning.

Step 3: Let the container reach Ollama#

On Linux, Ollama must listen on an address the container can reach.

Ollama listens on 127.0.0.1 by default. From inside a container on Linux, host.docker.internal points at the Docker bridge, not at loopback, so the call can be refused. If that happens, set OLLAMA_HOST (the variable is documented in the Ollama FAQ).

Linux#

For the standard systemd service:

sudo systemctl edit ollama

Add these lines in the editor, save, then restart:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

sudo systemctl restart ollama

Port 11434 is now open to your network, so block it from the LAN with your firewall. A safer alternative is to run Ollama as another service in the same Compose file and use api_base: http://ollama:11434.

Windows (PowerShell)#

Nothing to change if Ollama runs as the normal Windows app: Docker Desktop provides host.docker.internal. If the connection is refused, check that the Ollama app is running and that you pulled the model:

ollama list

Mac (Terminal)#

Same as Windows: Docker Desktop provides host.docker.internal. Check the model with:

ollama list

Step 4: Start LiteLLM and test it#

Start both containers, then list the models and send one chat request.

Linux#

docker compose up -d
docker compose logs --tail=50 litellm
set -a; . ./.env; set +a
curl -s http://127.0.0.1:4000/v1/models -H "Authorization: Bearer $LITELLM_MASTER_KEY"
curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"model":"local-llama","messages":[{"role":"user","content":"Say hello in five words."}]}'

Windows (PowerShell)#

docker compose up -d
docker compose logs --tail=50 litellm
$env:LITELLM_MASTER_KEY = ((Get-Content .env | Where-Object { $_ -like 'LITELLM_MASTER_KEY=*' }) -replace '^LITELLM_MASTER_KEY=','')
$h = @{ Authorization = "Bearer $env:LITELLM_MASTER_KEY" }
Invoke-RestMethod http://127.0.0.1:4000/v1/models -Headers $h | ConvertTo-Json -Depth 5
$body = @{ model = 'local-llama'; messages = @(@{ role = 'user'; content = 'Say hello in five words.' }) } | ConvertTo-Json -Depth 5
Invoke-RestMethod -Method Post http://127.0.0.1:4000/v1/chat/completions -Headers $h -ContentType 'application/json' -Body $body | ConvertTo-Json -Depth 8

Mac (Terminal)#

Same commands as Linux:

docker compose up -d
docker compose logs --tail=50 litellm
set -a; . ./.env; set +a
curl -s http://127.0.0.1:4000/v1/models -H "Authorization: Bearer $LITELLM_MASTER_KEY"
curl -s http://127.0.0.1:4000/v1/chat/completions \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"model":"local-llama","messages":[{"role":"user","content":"Say hello in five words."}]}'

Check it worked: the first call lists local-llama (and gpt-cloud) in a data array, and the second returns a standard OpenAI-style answer in choices[0].message.content. The first request to a local model can take a while because Ollama loads the model.

Any OpenAI SDK works the same way: set the base URL to http://127.0.0.1:4000 and use a key from LiteLLM.

Step 5: Create a virtual key with a budget#

Use the master key once to create a limited key, then give apps only that key.

Virtual keys, budgets and the Admin UI need the database. Without it you get the error No connected db..

Linux#

curl -s -X POST http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"key_alias":"laptop-test","models":["local-llama","gpt-cloud"],"max_budget":5,"budget_duration":"30d"}'

Windows (PowerShell)#

$k = @{ key_alias = 'laptop-test'; models = @('local-llama','gpt-cloud'); max_budget = 5; budget_duration = '30d' } | ConvertTo-Json -Depth 5
Invoke-RestMethod -Method Post http://127.0.0.1:4000/key/generate -Headers $h -ContentType 'application/json' -Body $k | ConvertTo-Json

Mac (Terminal)#

Same commands as Linux:

curl -s -X POST http://127.0.0.1:4000/key/generate \
  -H "Authorization: Bearer $LITELLM_MASTER_KEY" -H "Content-Type: application/json" \
  -d '{"key_alias":"laptop-test","models":["local-llama","gpt-cloud"],"max_budget":5,"budget_duration":"30d"}'

The answer contains a new key starting with sk-. Use it as the Bearer token in the chat request of Step 4. This key can only use the two listed models and is capped at a spend of 5 per 30 days. Duration units are s, m, h and d. The docs also describe several windows at once with budget_limits, for example [{"budget_duration":"24h","max_budget":2},{"budget_duration":"30d","max_budget":10}].

Local models are free, so tell LiteLLM. If you set input_cost_per_token: 0 and output_cost_per_token: 0 under the Ollama entry's litellm_params, budget checks are skipped for it, and a key that used up its cloud budget can still use the local model.

Admin UI: open http://127.0.0.1:4000/ui. The user name is admin and the password is your master key (or the values of UI_USERNAME and UI_PASSWORD if you set them).

Put it behind HTTPS with Traefik#

On a server with a domain, let Traefik publish LiteLLM over HTTPS and remove the host port.

This assumes Traefik v3 runs with the Docker provider, an entrypoint called https and a certificate for your hostname. Local-only users can skip this section. Create the network once if it is missing: docker network create proxy. In docker-compose.yml, delete the ports: block of litellm, then add the network and labels:

services:
  litellm:
    networks:
      - default
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.litellm.rule=Host(`litellm.<your-domain>`)"
      - "traefik.http.routers.litellm.entrypoints=https"
      - "traefik.http.routers.litellm.tls=true"
      - "traefik.http.services.litellm.loadbalancer.server.port=4000"
networks:
  proxy:
    external: true

Merge this into your existing file (the litellm service keeps its other settings; the default network keeps it connected to Postgres). Add traefik.http.routers.litellm.tls.certresolver=<your-resolver> if you use ACME. Behind a reverse proxy, the LiteLLM docs say to set general_settings.trusted_proxy_ranges to your proxy's address range; check the production page for the exact format. Keep Postgres off the proxy network.

Check it worked: curl -I https://litellm.<your-domain>/health/liveliness answers 200, and a request to /v1/models without a key is refused.

Security and testing#

The master key can do everything, so guard it and the network around it.

• Keep .env at chmod 600, out of git, and hand out virtual keys only. Keys created by an admin inherit admin rights, so create app keys with limited models and a budget.
• Give every key a budget and an expiry. The setting upperbound_key_generate_params in general_settings can cap what anyone can request.
• Do not publish port 4000 to the internet. Keep 127.0.0.1 or use HTTPS through Traefik. Keep Postgres and any Redis on private networks only.
• Back up LITELLM_SALT_KEY. Without it the stored provider keys cannot be read.
• Budgets are not a hard stop without the database, so also set spending limits on the provider side.
• Pin the image tag; do not use main-latest (no longer updated) or main-stable (deprecated).

Quick tests after every change:

docker compose ps
docker compose logs --tail=100 litellm
curl -s -o /dev/null -w '%{http_code}\n' http://127.0.0.1:4000/v1/models

The last line must print 401 (no key = no access).

Troubleshooting#

Diagnostics: docker compose logs --tail=200 litellm, docker compose ps, docker inspect. For more detail add --detailed_debug to the command: list.

No connected db.#

LiteLLM runs without a database, but virtual keys, budgets and the Admin UI do not. Make sure the db container is healthy and that DATABASE_URL is set in the litellm service.

ExceededTokenBudget#

The full text looks like ExceededTokenBudget: Current spend for token: ...; Max Budget for Token: .... Raise the limit with the /key/update endpoint or wait for the budget period to reset.

Updates and backups#

Back up the database, the config and the secrets, then update by pinning a new tag.

What to back up: a Postgres dump, config/config.yaml, and .env (it holds the salt key; without the salt key a restored database cannot decrypt stored provider keys). Store the backup copy of .env somewhere private and encrypted.

Linux#

cd ~/docker_volumes/litellm
D=$(date +%F)
docker compose exec -T db pg_dump -U llmproxy -Fc litellm > "backups/litellm-$D.dump"
tar czf "backups/litellm-config-$D.tar.gz" config .env
chmod 600 backups/*
# restore test into a NEW database, never over the live one:
docker compose exec -T db createdb -U llmproxy litellm_restore
docker compose exec -T db pg_restore -U llmproxy -d litellm_restore < "backups/litellm-$D.dump"
docker compose exec -T db psql -U llmproxy -d litellm_restore -c '\dt' | head
docker compose exec -T db dropdb -U llmproxy litellm_restore

Schedule the first four lines in a script with cron at 03:00 (0 3 * * *).

Windows (PowerShell)#

Set-Location "$HOME\docker_volumes\litellm"
$d = Get-Date -Format yyyy-MM-dd
docker compose exec -T db pg_dump -U llmproxy -Fc -f /tmp/litellm.dump litellm
docker compose cp db:/tmp/litellm.dump "backups\litellm-$d.dump"
Compress-Archive -Path config, .env -DestinationPath "backups\litellm-config-$d.zip"

Run it daily with Task Scheduler (schtasks /Create /SC DAILY /ST 03:00 ...). Restore test: docker compose exec -T db createdb -U llmproxy litellm_restore, run pg_restore -U llmproxy -d litellm_restore /tmp/litellm.dump in the container (the dump is still there after the backup above), then drop the test database with dropdb.

Mac (Terminal)#

Same commands as Linux. Schedule with cron or launchd.

Updating: read the release notes, change the tag in docker-compose.yml to the new vX.Y.Z, then:

docker compose pull litellm
docker compose up -d litellm
docker compose logs --tail=100 litellm

Take the backup first. A new version may change the database schema when it starts, so going back to an older tag can mean restoring the dump. Stable tags are plain vX.Y.Z now.

Performance and resources#

The proxy itself is light; model speed comes from Ollama or the provider, so measure rather than guess.

• Watch memory and CPU with docker stats.
• For production the docs recommend turning off verbose logging, using JSON logs and a request timeout, for example under litellm_settings: request_timeout: 600 and json_logs: true. Setting LITELLM_LOG: "ERROR" and LITELLM_MODE: "PRODUCTION" in the container environment reduces noise; production mode disables loading a .env file, which is fine here because Compose passes the variables in.
• Local models are limited by your RAM or GPU, not by LiteLLM. Pick a model size that fits (see ollama ps).
• The first call to a local model is slow while Ollama loads it; later calls are faster.

FAQ#

Do I need Postgres? Not for a plain API in front of models. You need it for virtual keys, budgets, spend tracking and the Admin UI.

Which image should I use? ghcr.io/berriai/litellm:v1.106.3 or a newer pinned vX.Y.Z. It contains what the database setup needs.

Why can't the container see Ollama? localhost inside the container is the container. Use http://host.docker.internal:11434, and on Linux let Ollama listen on a reachable address (Step 3).

What is the difference between ollama/ and ollama_chat/? ollama_chat/ uses the /api/chat endpoint and is the recommended one for chat models.

Can I put a budget on local models? Yes, but they cost nothing. Set both token costs to 0 so budget checks are skipped for them.

Sources#

• LiteLLM Docker quick start
• Virtual keys, users and budgets, Admin UI
• Going to production, deployment, image security
• Ollama provider
• LiteLLM releases (v1.106.3, 24 October 2026) and container images
• Ollama FAQ (OLLAMA_HOST)
• Checked against the official docs on 2026-10-26; not run by DIY Lab Hub for this guide.