LocalAI with Docker: a Drop-in OpenAI API for Text, Images and Audio on Your Own Hardware (Linux, Windows, Mac)

What you get in one minute#

LocalAI is a free, open-source server that speaks the OpenAI API but runs on your own machine, so apps built for OpenAI can talk to your own models for chat, embeddings, images and speech.

Logo: the official LocalAI project (github.com/mudler/LocalAI)

You change one thing in your app: the base URL. Instead of https://api.openai.com/v1 it uses http://localhost:8080/v1. Your prompts and files stay on your hardware. LocalAI also has a built-in web page where you chat, install models and watch what is loaded.

At a glance#

• What you get: one API address for text chat, embeddings, image generation and audio (speech-to-text and text-to-speech), plus a web interface.
• Time needed: about 20 minutes, plus the model download.
• Difficulty: easy to medium. The GPU image choice is the only tricky part.
• You need: Docker with the Compose plugin, 8 GB of RAM or more, and 10 GB of free disk for a first small model. A GPU is optional.

Why this version? We checked the open issues created since 1 October 2026 on GitHub. We found no wave of install failures or data-loss reports, only narrow bugs (listed under Known bugs). The latest release is a small security patch, so this guide uses it. The previous release is v4.12.0, your rollback tag. The Perplexity research for this guide did not complete, so every fact below comes from the official docs at the v4.12.1 tag.

How it fits together#

Your app sends an OpenAI-style request to port 8080, LocalAI picks the right engine (backend) for the job and loads the model files from your disk.

Good to know: the core image is small. Each engine is a separate backend that LocalAI downloads the first time a model needs it, and it picks one that matches your GPU by itself. So the first install of a model takes longer than the next ones.

Before you start#

You need Docker, free port 8080 and enough RAM for the model you choose.

Check your tools:

docker --version
docker compose version

Checklist:

• [ ] Port 8080 is free on this machine
• [ ] You have 10 GB or more free disk (models of 1-3 billion parameters are 1-3 GB, 7-13 billion are 4-8 GB)
• [ ] You backed up any config you are about to replace

Tested on 2026-10-26: not run. Commands checked against the official LocalAI docs at the v4.12.1 tag.

Which image for which hardware#

Pick the image tag that matches your graphics card, not the one that sounds newest. The tag goes after localai/localai:. This guide pins the version, so write v4.12.1 instead of latest.

We checked on Docker Hub that all of the tags in the first six rows exist for v4.12.1. The Jetson tag follows the official naming (not checked).

What runs on a Mac, honestly#

• Docker on a Mac has no GPU access. Containers run inside a Linux virtual machine, and none of the GPU images can use the Apple chip. The CPU image has an arm64 build, so it starts, but it is slow for big models.
• The official Mac way is the native app. The docs offer a LocalAI.dmg: a small menu-bar launcher that downloads and runs the LocalAI server directly on macOS, signed and notarized by Apple. You install it by dragging it to Applications. The docs do not say which Apple GPU features it uses (VERIFY with a small model and Activity Monitor).
• So on a Mac: use the DMG app for real work, and use Docker only to test the compose file or the API.

Windows#

• CPU image: always works through Docker Desktop.
• NVIDIA: Docker Desktop can pass an NVIDIA card into WSL2 containers when the NVIDIA Windows driver is current. Use the CUDA 12 image and --gpus all.
• AMD and Intel images need Linux device files (/dev/kfd, /dev/dri) that Docker Desktop does not provide. On Windows, use the CPU image or an NVIDIA card.

Step 1: Create the folders and the key#

You create one folder tree under docker_volumes and one secret API key. LocalAI wants four folders: models, backends, configuration and data. If you skip one, you lose it when you upgrade the container.

Linux#

mkdir -p ~/docker_volumes/localai/{models,backends,configuration,data,backups}
cd ~/docker_volumes/localai
echo "LOCALAI_API_KEY=$(openssl rand -hex 24)" > .env
chmod 600 .env

Windows (PowerShell)#

$d = "$HOME\docker_volumes\localai"
"models","backends","configuration","data","backups" | ForEach-Object { New-Item -ItemType Directory -Force "$d\$_" | Out-Null }
Set-Location $d
$b = New-Object byte[] 24
[Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($b)
"LOCALAI_API_KEY=" + (($b | ForEach-Object { $_.ToString('x2') }) -join '') | Set-Content -Encoding ascii .env

Mac (Terminal)#

Same commands as Linux:

mkdir -p ~/docker_volumes/localai/{models,backends,configuration,data,backups}
cd ~/docker_volumes/localai
echo "LOCALAI_API_KEY=$(openssl rand -hex 24)" > .env
chmod 600 .env

Check it worked#

ls ~/docker_volumes/localai

Expected: the four folders, backups and a hidden .env file (use ls -a). Do not paste the key in chats or screenshots.

Step 2: Write the compose file and start LocalAI#

One service, one pinned image, four volumes. The port is bound to 127.0.0.1, so only this machine can reach it until you add HTTPS.

Save this as docker-compose.yml in the same folder (Linux and Mac: nano docker-compose.yml; Windows: notepad docker-compose.yml):

services:
  localai:
    image: localai/localai:v4.12.1
    container_name: localai
    restart: unless-stopped
    ports:
      - "127.0.0.1:8080:8080"
    environment:
      - LOCALAI_API_KEY=${LOCALAI_API_KEY}
    volumes:
      - ./models:/models
      - ./backends:/backends
      - ./configuration:/configuration
      - ./data:/data
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:8080/readyz"]
      start_period: 60m
      interval: 1m
      timeout: 10s
      retries: 3

The health check comes from the official docs. Its long start_period is on purpose: the first start can download a lot before the API answers.

For an NVIDIA card (Linux, or Windows with WSL2) change the image to localai/localai:v4.12.1-gpu-nvidia-cuda-12 and add this block to the service. It is the "CDI" form the docs recommend for NVIDIA Container Toolkit 1.14 and newer:

    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia.com/gpu
              count: all
              capabilities: [gpu]

Older toolkit (before 1.14)? Use driver: nvidia, count: 1 and capabilities: [gpu] instead.

For AMD or Intel (Linux) change the image to the tag from the table and add the devices:

    devices:
      - /dev/kfd
      - /dev/dri
    group_add:
      - video

For Intel, list only /dev/dri. Now start it:

Linux#

docker compose up -d

Windows (PowerShell)#

docker compose up -d

Mac (Terminal)#

Same command as Linux (CPU only inside Docker, see above):

docker compose up -d

Check it worked#

docker compose ps
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/readyz

Expected: the container is running (healthy after the first start finishes) and the second command prints 200. On Windows use curl.exe instead of curl. Then open http://localhost:8080 in your browser. The web interface asks for your API key: it is the value in .env.

First run: install a model#

Install a small model from the gallery first, so you can see the whole chain work. The docs use qwen3-4b: small, CPU-friendly and able to call tools.

1. Open http://localhost:8080 and enter your API key if asked.
2. Open Models, then Explore, and search for qwen3-4b.
3. Click Install and wait until the download finishes.
4. Open Chat, choose qwen3-4b and send a message.

Prefer the terminal? Run the install inside the container (same on all three systems; the API key is needed only for the web and HTTP calls):

docker exec localai local-ai models install qwen3-4b

See what is available with docker exec localai local-ai models list.

Check it worked#

The chat page answers within a few seconds on a modern CPU for a small model. Check the logs if nothing happens:

docker logs localai --tail 50

Step 3: Call the API (chat, embeddings, images, audio)#

Every call uses the standard OpenAI address with your key as a bearer token. Each example needs a model of that kind to be installed first.

Linux#

Load the key into your shell, then chat:

export LOCALAI_API_KEY=$(grep ^LOCALAI_API_KEY= ~/docker_volumes/localai/.env | cut -d= -f2)

curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-4b","messages":[{"role":"user","content":"Say hello in five words."}]}'

Embeddings (install qwen3-embedding-0.6b first with docker exec localai local-ai models install qwen3-embedding-0.6b):

curl http://localhost:8080/v1/embeddings \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-embedding-0.6b","input":"My text to embed"}'

Image generation. The docs install Stable Diffusion at runtime through /models/apply:

curl http://localhost:8080/models/apply \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"github:mudler/LocalAI/gallery/stablediffusion.yaml@master"}'

curl http://localhost:8080/v1/images/generations \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"prompt":"A cute baby sea otter","size":"256x256"}'

Speech to text with Whisper (the docs name the model whisper-1, so OpenAI clients work unchanged):

curl http://localhost:8080/models/apply \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"url":"github:mudler/LocalAI/gallery/whisper-base.yaml@master","name":"whisper-1"}'

curl http://localhost:8080/v1/audio/transcriptions \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -F file="@$PWD/sample.ogg" -F model="whisper-1"

Text to speech uses POST /v1/audio/speech with a model, an input text and a voice. Install a text-to-speech model from the gallery first and use its name.

Windows (PowerShell)#

PowerShell quotes JSON badly in curl, so use Invoke-RestMethod:

$key = ((Get-Content "$HOME\docker_volumes\localai\.env") -replace '^LOCALAI_API_KEY=','')
$h = @{ Authorization = "Bearer $key" }

$body = @{ model = "qwen3-4b"; messages = @(@{ role = "user"; content = "Say hello in five words." }) } | ConvertTo-Json -Depth 5
Invoke-RestMethod -Uri http://localhost:8080/v1/chat/completions -Method Post -Headers $h -ContentType "application/json" -Body $body

$emb = @{ model = "qwen3-embedding-0.6b"; input = "My text to embed" } | ConvertTo-Json
Invoke-RestMethod -Uri http://localhost:8080/v1/embeddings -Method Post -Headers $h -ContentType "application/json" -Body $emb

$img = @{ prompt = "A cute baby sea otter"; size = "256x256" } | ConvertTo-Json
Invoke-RestMethod -Uri http://localhost:8080/v1/images/generations -Method Post -Headers $h -ContentType "application/json" -Body $img

Audio upload works best with the real curl.exe:

curl.exe http://localhost:8080/v1/audio/transcriptions -H "Authorization: Bearer $key" -F "[email protected]" -F "model=whisper-1"

Install the image and Whisper models with the same /models/apply bodies as on Linux, sent with Invoke-RestMethod, or click them in Models in the web interface.

Mac (Terminal)#

Same commands as Linux:

export LOCALAI_API_KEY=$(grep ^LOCALAI_API_KEY= ~/docker_volumes/localai/.env | cut -d= -f2)
curl http://localhost:8080/v1/chat/completions \
  -H "Authorization: Bearer $LOCALAI_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3-4b","messages":[{"role":"user","content":"Say hello in five words."}]}'

Using the native Mac app instead of Docker? The API address is the same, http://localhost:8080.

Check it worked#

Expected from the chat call: a JSON answer with choices[0].message.content holding the reply. A wrong or missing key gives an authentication error instead (see Troubleshooting). Use LocalAI from an OpenAI library by setting the base URL to http://localhost:8080/v1 and the API key to your key.

Using it for RAG#

Embeddings are the building block for chat with your documents. Install an embedding model, point your RAG tool (AnythingLLM, Open WebUI, Dify) at http://localhost:8080/v1 and choose the embedding model by name. The docs say that if a request has no model, LocalAI uses the embedding model whose name comes first alphabetically, so always send the model field. If no embedding model is installed, the request returns 400 Bad Request.

How LocalAI compares#

LocalAI is the "one server for everything" option. Ollama is the easiest for chat. vLLM is the fastest for many users at once on NVIDIA.

The LocalAI column comes from the official docs. The Ollama and vLLM columns are our general summary, not official claims. For a smaller, text-only server see our llama.cpp server guide.

Put it behind HTTPS (with or without Traefik)#

With Traefik#

This is for a server with a domain name and Traefik already running. Running only at home on localhost? Skip this section.

Prerequisites: a domain, DNS pointing to the server, ports 80 and 443 open, and Traefik on the external proxy network.

docker network create proxy

Change the service like this: drop the ports: block and add the network and labels. The labels are our example, not from the official docs (VERIFY against your Traefik setup):

  localai:
    image: localai/localai:v4.12.1
    container_name: localai
    restart: unless-stopped
    environment:
      - LOCALAI_API_KEY=${LOCALAI_API_KEY}
    volumes:
      - ./models:/models
      - ./backends:/backends
      - ./configuration:/configuration
      - ./data:/data
    networks:
      - proxy
    labels:
      - traefik.enable=true
      - traefik.docker.network=proxy
      - traefik.http.routers.localai.rule=Host(`<your-domain>`)
      - traefik.http.routers.localai.entrypoints=https
      - traefik.http.routers.localai.tls=true
      - traefik.http.services.localai.loadbalancer.server.port=8080

networks:
  proxy:
    external: true

Without Traefik: Caddy#

Choose Caddy if you have no Traefik yet and run one small server.

Caddyfile for a public domain (LocalAI keeps its 127.0.0.1:8080 port):

localai.example.com {
    reverse_proxy localhost:8080
}

Caddyfile for home only (point the name at the Caddy machine):

localai.home.arpa {
    tls internal
    reverse_proxy localhost:8080
}

Trust the root certificate (pki/authorities/local/root.crt in Caddy's data directory):

# Linux: on the Caddy machine
sudo caddy trust
# other Linux devices, with root.crt copied over
sudo cp root.crt /usr/local/share/ca-certificates/caddy-root.crt && sudo update-ca-certificates
# Windows (PowerShell as administrator)
certutil -addstore -f ROOT root.crt
# Mac
sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain root.crt

Run Caddy (Linux: install from the official apt repository on caddyserver.com/docs/install first):

# Linux
sudo systemctl reload caddy
# Windows (PowerShell)
winget install CaddyServer.Caddy
caddy run
# Mac
brew install caddy && caddy run

Or use Docker on the app's network. Add the proxy network to the localai service, use reverse_proxy localai:8080 in the Caddyfile and save this in ~/docker_volumes/caddy (create the network once: docker network create proxy):

services:
  caddy:
    image: caddy:2.11.7
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./caddy_data:/data
      - ./caddy_config:/config
    networks:
      - proxy
networks:
  proxy:
    external: true

Streaming answers (server-sent events) work through Caddy's reverse_proxy without extra settings. Check it worked: curl -I https://localai.example.com answers HTTP/2 with a status code (a call without a key returns an authentication error, which proves the key gate works; with -H "Authorization: Bearer <YOUR_KEY>" on /v1/models it answers 200).

Security and testing#

An OpenAI-compatible server without a key is an open door to your CPU and GPU. Keep it private and keep the key set.

• Keep the port on localhost. The compose file binds 127.0.0.1:8080. Do not forward 8080 to the internet. Reach it from outside through WireGuard or Tailscale, or through the HTTPS section above.
• Key in .env. chmod 600 .env (Linux/Mac). LocalAI accepts the key as Authorization: Bearer, x-api-key, xi-api-key or a token cookie. Several keys can be comma-separated.
• The simple key is an admin key. The docs say API keys grant full admin access with no role separation. For several users set LOCALAI_AUTH=true: it adds admin and user roles, OAuth login, per-user keys and usage tracking (read the official authentication page first; it stores users in a database in your data folder).
• Models are code you download. The docs warn to be careful with models from untrusted sources. Use the gallery or well-known publishers.
• v4.12.1 is a security fix. It hardens how LocalAI fetches URLs you send it (vision and audio URLs). Another reason to update.

Tests:

docker compose ps
docker logs localai --tail 100
curl -s -o /dev/null -w "%{http_code}\n" http://localhost:8080/v1/models
curl -s -o /dev/null -w "%{http_code}\n" -H "Authorization: Bearer $LOCALAI_API_KEY" http://localhost:8080/v1/models

Expected: the container is running, and the last two commands print an authentication error code without the key and 200 with it.

Troubleshooting#

Exact error: NVIDIA legacy mode#

nvidia-container-cli: requirement error: invalid expression
Auto-detected mode as 'legacy'

The docs say this is a Docker and NVIDIA Container Toolkit problem, not a LocalAI bug. Fix: use the CDI block (driver: nvidia.com/gpu) from Step 2, upgrade the toolkit to 1.14 or newer, and check nvidia-container-cli --query-gpu on the host.

Known bugs#

Open on GitHub at the time of writing (read 2026-10-26):

• #12893: backend install fails with mkdir //downloading: permission denied on some setups. Check that the backends folder is writable.
• #12735: a Python backend fails with libmpi_cxx.so.40: cannot open shared object file.
• #12924: the audio-cpp-stable-audio-sfx install fails with a checksum mismatch.
• #12806: one gallery model fails to download with a 404.
• If v4.12.1 misbehaves for you, roll back by setting the tag to v4.12.0.

Updates and backups#

Update by changing the tag. Back up the settings and data, and decide if you also want the big model files.

Update (Linux/Mac; on Windows use the same docker compose lines in PowerShell):

cd ~/docker_volumes/localai
# read the release notes, edit the image tag, then:
docker compose pull && docker compose up -d

Backup script (Linux/Mac). Settings and data are small. Models can be downloaded again, so leave them out unless your internet is slow:

cd ~/docker_volumes/localai
docker compose stop localai
tar czf backups/localai-$(date +%F).tgz configuration data .env docker-compose.yml
docker compose start localai

Add models to the tar line if you want them too. Schedule: Linux/Mac cron at 03:00 (0 3 * * * <path-to-script>). Windows: Task Scheduler running the same steps in PowerShell with Compress-Archive.

Restore test#

Do this once, before you need it:

mkdir -p /tmp/restore && tar xzf ~/docker_volumes/localai/backups/localai-*.tgz -C /tmp/restore
ls /tmp/restore/configuration /tmp/restore/data
rm -rf /tmp/restore

Expected: your settings and data files are listed. To test a full restore, extract into a new folder, start a copy of the compose file there on another port (for example 127.0.0.1:8081:8080) and open it.

Performance and resources#

Speed depends on your model and your hardware, not on LocalAI. We did not measure RAM or speed ourselves, and the docs give no fixed numbers, so measure on your machine:

docker stats --no-stream localai

Tips from the docs and common sense:

• Keep the model smaller than your RAM (or GPU memory). A model bigger than your RAM runs poorly.
• Use an SSD, and keep 2-3 times the model size free while downloading.
• Quantized files are smaller: Q4_K_M is about 75% of the original size and Q2_K about 50%, with some loss of quality.
• Match the image to the card. A CUDA image on an NVIDIA card is much faster than the CPU image.
• Only the backends your models use are installed, so the folder stays smaller than you fear.

Bigger setups: LocalAI also has a distributed mode (worker nodes that register with a frontend) and peer-to-peer federation. Both are in the official docs and the web interface (Nodes page). We do not cover them here.

FAQ#

Is LocalAI really a drop-in for OpenAI? For the common calls, yes: chat, embeddings, images and audio use the same paths (/v1/chat/completions, /v1/embeddings, /v1/images/generations, /v1/audio/transcriptions). It also offers the Anthropic Messages API and the Open Responses API. Model names are your own, though, and quality depends on the model.

Can I run it without a GPU? Yes. The CPU image works everywhere. Pick a small model such as qwen3-4b.

What about my Mac? Docker on a Mac has no GPU, so use the native LocalAI app (LocalAI.dmg) for real work and Docker only for testing.

Can I use my own models? Yes. Install gallery models by name, pull a GGUF from Hugging Face with local-ai run huggingface://..., or drop files and a YAML config into the models folder.

Where are my models stored? In ~/docker_volumes/localai/models in this guide (/models inside the container).

Is it safe to expose to the internet? Not as it is. Keep the key, bind to localhost, and use HTTPS or a VPN. Read the authentication page of the official docs before you let other people in.

Sources#

• LocalAI quickstart and container images (docs at the v4.12.1 tag, read 2026-10-26)
• Try it out: API examples, embeddings, image generation, audio to text (read 2026-10-26)
• macOS installation and authentication (read 2026-10-26)
• LocalAI on GitHub: releases and issues (MIT licence, read 2026-10-26)
• Docker Hub: localai/localai tags (read 2026-10-26)
• Caddy documentation (automatic HTTPS, tls internal, reverse_proxy, install)

Checked against the official docs on 2026-10-26.