Text Embeddings Inference (TEI) with Docker: a local embedding and reranker server for RAG (Linux, Windows, Mac)

What you get in one minute#

Text Embeddings Inference (TEI) is Hugging Face's open-source server that turns text into embedding vectors, and scores search results with a reranker, through a small HTTP API, so your RAG tools can use a fast model on your own machine instead of a cloud service.

Logo: the official Hugging Face brand assets (huggingface.co/brand), the company behind TEI (github.com/huggingface/text-embeddings-inference)

A RAG tool makes embeddings twice: when it indexes your documents and for every question. TEI does only that job: it loads one model, batches requests and answers quickly, on CPU or an NVIDIA GPU. It also serves reranker models, which reorder search hits so the best comes first. The project is alive: not archived and not renamed, with release v1.9.4 in September 2026 and a commit in November 2026.

At a glance:

• What it is: one container, one model, plain HTTP. /embed and an OpenAI-style /v1/embeddings make vectors; /rerank scores texts against a question.
• CPU first: the CPU image needs no GPU. NVIDIA GPUs are supported with a separate image per GPU generation.
• Pin the tag: use cpu-1.9.4, not latest. Pick the GPU image by your card, not by a generic name.
• One vector size per index: the model fixes the number of dimensions (384 for our starter model). Change the model and you must index again.
• Rerank has its own format: Open WebUI and LightRAG send a different rerank request, so TEI's /rerank does not plug into them directly (see "Connect your RAG tool").

How it fits together#

Your RAG tool sends text to TEI and gets vectors back. The tool, not TEI, stores the vectors in a vector database and searches them. On the first start TEI downloads the model once and keeps it in a folder on your disk.

Drawn for this guide from the official README, API reference and Dockerfile (v1.9.4, November 2026). The dashed reranker is optional.

Our LightRAG guide uses Ollama for embeddings, which is fine for a small setup (see our Ollama guide). Pick TEI when you want a server that only does embeddings and reranking and offers a /rerank endpoint.

Before you start#

Check the tools (on Windows PowerShell use curl.exe):

docker --version
docker compose version
curl --version

• [ ] Docker is running and docker compose version prints a v2 version
• [ ] Port 8080 is free on the host (ss -ltn on Linux, netstat -ano on Windows)
• [ ] You know your CPU type: uname -m prints x86_64 or aarch64 (arm64) on Linux and Mac
• [ ] For the HTTPS section only: a domain name, DNS pointing at the server, ports 80 and 443 free

Tested by DIY Lab Hub: not yet. The commands come from the official README, docs, OpenAPI file, Dockerfile and registry tags at v1.9.4, read in November 2026. We did not start the containers; where we show output, it is the shape the official API reference describes.

Step 1: Create the folders and the secret file#

TEI keeps downloaded models in /data inside the container. We map that to ~/docker_volumes/tei/data, as all DIY Lab Hub guides do. The .env file holds the image tag and an API key that every request must send.

Linux#

mkdir -p ~/docker_volumes/tei/{data,backups}
cd ~/docker_volumes/tei
printf 'TEI_TAG=cpu-1.9.4\nTEI_API_KEY=%s\n' "$(openssl rand -hex 32)" > .env
chmod 600 .env

If uname -m printed aarch64, change the tag to cpu-arm64-1.9.4 in .env.

Windows (PowerShell)#

New-Item -ItemType Directory -Force "$HOME\docker_volumes\tei\data", "$HOME\docker_volumes\tei\backups" | Out-Null
Set-Location "$HOME\docker_volumes\tei"
$bytes = New-Object byte[] 32
[Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($bytes)
$key = -join ($bytes | ForEach-Object { $_.ToString('x2') })
Set-Content -Path .env -Value "TEI_TAG=cpu-1.9.4`nTEI_API_KEY=$key" -Encoding ascii

Mac (Terminal)#

Same commands as Linux. On an Apple Silicon Mac, set TEI_TAG=cpu-arm64-1.9.4 in .env; on an Intel Mac keep cpu-1.9.4.

Check it worked. On Linux and Mac, ls -l .env shows -rw-------. On Windows, Test-Path .env prints True. Do not print the file in a shared screen: it holds your key.

Tip: Make the key now, even for a home setup. TEI answers every request when no key is set (docs: "By default the server responds to every request").

Step 2: Write the Compose file and start TEI#

The file below pins the image by the tag in .env, publishes the API on 127.0.0.1 only, keeps the model cache in ./data and checks /health. The reranker is a second service that starts only when you ask for it. Both join an external network called proxy, so other containers (your RAG tool, a reverse proxy) can reach them by name.

Save it as docker-compose.yml in the TEI folder. On Linux and Mac use nano docker-compose.yml; on Windows use notepad docker-compose.yml.

services:
  tei:
    image: ghcr.io/huggingface/text-embeddings-inference:${TEI_TAG}
    container_name: tei
    restart: unless-stopped
    command: ["--model-id", "BAAI/bge-small-en-v1.5"]
    environment:
      - API_KEY=${TEI_API_KEY}
    ports:
      - "127.0.0.1:8080:80"
    volumes:
      - ./data:/data
    networks:
      - default
      - proxy
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:80/health"]
      interval: 30s
      timeout: 5s
      retries: 5
      start_period: 120s

  tei-rerank:
    image: ghcr.io/huggingface/text-embeddings-inference:${TEI_TAG}
    container_name: tei-rerank
    restart: unless-stopped
    profiles: ["rerank"]
    command: ["--model-id", "BAAI/bge-reranker-base"]
    environment:
      - API_KEY=${TEI_API_KEY}
    ports:
      - "127.0.0.1:8081:80"
    volumes:
      - ./data:/data
    networks:
      - default
      - proxy

networks:
  proxy:
    external: true

The model ID BAAI/bge-small-en-v1.5 is a small English model (33 million parameters, MIT licence). The command replaces the image's default --json-output, so logs are plain text. Now create the network and start:

Linux#

cd ~/docker_volumes/tei
docker network create proxy
docker compose up -d
docker compose ps
docker compose logs --tail=50 tei

If docker network create says the network already exists, that is fine.

Windows (PowerShell)#

Set-Location "$HOME\docker_volumes\tei"
docker network create proxy
docker compose up -d
docker compose ps
docker compose logs --tail=50 tei

Mac (Terminal)#

Same commands as Linux.

Check it worked. The first start downloads the model from huggingface.co, so wait. docker compose ps shows tei as healthy when the model has loaded (the health check starts counting after 120 seconds). The docs publish no expected log text, so read the log for errors rather than for a fixed line.

Step 3: Test the embedding API#

/health needs no key; every other route needs Authorization: Bearer <key>. The model's own config lists 384 dimensions, so a vector should have 384 numbers.

Linux#

cd ~/docker_volumes/tei
KEY=$(grep '^TEI_API_KEY=' .env | cut -d= -f2)
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8080/health
curl -s -o /dev/null -w "%{http_code}\n" http://127.0.0.1:8080/info
curl -s http://127.0.0.1:8080/info -H "Authorization: Bearer $KEY"
curl -s http://127.0.0.1:8080/embed -X POST \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"inputs":"What is Deep Learning?"}' | head -c 100
curl -s http://127.0.0.1:8080/v1/embeddings -X POST \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"input":"What is Deep Learning?","model":"text-embeddings-inference"}' \
  | python3 -c "import sys,json; print(len(json.load(sys.stdin)['data'][0]['embedding']))"

Windows (PowerShell)#

Set-Location "$HOME\docker_volumes\tei"
$key = ((Get-Content .env) | Where-Object { $_ -like 'TEI_API_KEY=*' }) -replace '^TEI_API_KEY=', ''
$h = @{ Authorization = "Bearer $key" }
(Invoke-WebRequest http://127.0.0.1:8080/health -UseBasicParsing).StatusCode
Invoke-RestMethod http://127.0.0.1:8080/info -Headers $h
$body = '{"input":"What is Deep Learning?","model":"text-embeddings-inference"}'
$r = Invoke-RestMethod -Uri http://127.0.0.1:8080/v1/embeddings -Method Post -Headers $h -ContentType 'application/json' -Body $body
$r.data[0].embedding.Count

Mac (Terminal)#

Same commands as Linux.

Check it worked. Expected, from the official API reference (we did not capture a run):

200            <- /health
401            <- /info without the key
{"model_id":"BAAI/bge-small-en-v1.5", ... "max_input_length": ..., "max_client_batch_size":32, ...}
[[ ...a list holding one list of numbers... ]]
384            <- length of the vector

/info lists the model, max_input_length, max_batch_tokens and max_client_batch_size. Write down max_input_length: it is the longest text TEI embeds in full.

Tip: Send {"inputs": ...} to /embed and {"input": ...} to /v1/embeddings. The field names differ by one letter, and a wrong one gives a validation error, not a vector.

Step 4: Optional, add the reranker#

A reranker reads a question and a list of texts together and scores how well each answers it. It is slower than vector search but more exact, so run it on the top hits only. BAAI/bge-reranker-base is MIT licensed and about 1.1 GB.

Linux#

cd ~/docker_volumes/tei
docker compose --profile rerank up -d
KEY=$(grep '^TEI_API_KEY=' .env | cut -d= -f2)
curl -s http://127.0.0.1:8081/rerank -X POST \
  -H "Authorization: Bearer $KEY" -H 'Content-Type: application/json' \
  -d '{"query":"What is Deep Learning?","texts":["Deep Learning is not...","Deep learning is..."]}'

Windows (PowerShell)#

Set-Location "$HOME\docker_volumes\tei"
docker compose --profile rerank up -d
$key = ((Get-Content .env) | Where-Object { $_ -like 'TEI_API_KEY=*' }) -replace '^TEI_API_KEY=', ''
$body = '{"query":"What is Deep Learning?","texts":["Deep Learning is not...","Deep learning is..."]}'
Invoke-RestMethod -Uri http://127.0.0.1:8081/rerank -Method Post -Headers @{ Authorization = "Bearer $key" } -ContentType 'application/json' -Body $body

Mac (Terminal)#

Same commands as Linux.

Check it worked. The answer is a list of objects with index (the position of the text you sent) and score. In the v1.9.4 source the list is sorted with the best score first. Use index to map each score back to your text. The first call waits for the model download.

Optional: use an NVIDIA GPU (Linux)#

The TEI docs give one image per GPU generation. Using the wrong one is the top source of GPU trouble, so look up your card:

Install the NVIDIA Container Toolkit first; our NVIDIA GPU guide shows how. Then save this as docker-compose.gpu.yml next to docker-compose.yml. It follows the GPU block in RAGFlow's official compose file:

services:
  tei:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

Start with docker compose -f docker-compose.yml -f docker-compose.gpu.yml up -d. Check the log for the CUDA device. Windows and Mac: not covered here.

Mac option: Metal without Docker. On Apple Silicon, the docs install a prebuilt binary and use the GPU through Metal. We did not test it:

brew install text-embeddings-inference
text-embeddings-router --model-id BAAI/bge-small-en-v1.5 --port 8080 --api-key <your-key>

Picking a model#

TEI serves models that carry the text-embeddings-inference tag on the Hugging Face Hub. We read these model pages on 5 November 2026:

TEI's Apache-2.0 licence does not cover the models: read each model's licence. A gated model (for example google/embeddinggemma-300m) needs a Hugging Face account, accepted terms and a read token: put HF_TOKEN=<your read token> in the environment: list.

To change the model, edit --model-id in docker-compose.yml, run docker compose up -d, and index again: vectors from two models cannot be mixed.

Connect your RAG tool#

All three tools below join the same proxy network as TEI, so they reach it by container name: http://tei:80. We read how each tool calls the API in its docs or source; we did not run the combinations.

Open WebUI#

Our Open WebUI guide covers the install. Add the proxy network to its service, then these variables. Open WebUI posts to <base>/embeddings, so the base URL ends in /v1:

    environment:
      - RAG_EMBEDDING_ENGINE=openai
      - RAG_OPENAI_API_BASE_URL=http://tei:80/v1
      - RAG_OPENAI_API_KEY=<your TEI_API_KEY>
      - RAG_EMBEDDING_MODEL=text-embeddings-inference
      - RAG_EMBEDDING_BATCH_SIZE=16

After the first start the value saved in the admin panel wins over the environment (ENABLE_PERSISTENT_CONFIG is on by default). Keep the batch size at or below TEI's limit of 32.

LightRAG#

Our LightRAG guide uses Ollama; to switch, set these in its .env before the first document (the vector size is fixed after that):

EMBEDDING_BINDING=openai
EMBEDDING_BINDING_HOST=http://tei:80/v1
EMBEDDING_BINDING_API_KEY=<your TEI_API_KEY>
EMBEDDING_MODEL=BAAI/bge-small-en-v1.5
EMBEDDING_DIM=384
EMBEDDING_BATCH_NUM=32

LightRAG's default CHUNK_SIZE is 1200 tokens, more than the 512 positions the bge-small config lists. Lower it, or use Qwen3-Embedding-0.6B (see "Common mistakes").

RAGFlow#

Our RAGFlow guide uses Ollama. RAGFlow's own compose files ship a tei-cpu and a tei-gpu profile with a TEI image; its model list also has a HuggingFace provider, and in its source that provider posts {"inputs": ...} to <base_url>/embed and {"query": ..., "texts": ...} to <base_url>/rerank: exactly TEI's format. Two catches from the source: it sends no Authorization header, and it waits 30 seconds per call. So for RAGFlow, remove the API_KEY line from the TEI service, keep TEI off the internet, and publish nothing beyond 127.0.0.1.

Qdrant, by script#

TEI makes the vectors; your tool stores them. Our Qdrant guide runs the database. This script is our own sketch, not from the TEI or Qdrant docs. It embeds three sentences, stores them in a new collection and asks which one answers a question, using only the standard library:

import json
import os
import urllib.request

TEI = os.environ.get("TEI_URL", "http://127.0.0.1:8080")
TEI_KEY = os.environ.get("TEI_API_KEY", "")
QDRANT = os.environ.get("QDRANT_URL", "http://127.0.0.1:6333")
QDRANT_KEY = os.environ.get("QDRANT_API_KEY", "")


def call(method, url, body, headers):
    req = urllib.request.Request(url, data=json.dumps(body).encode(), method=method)
    req.add_header("Content-Type", "application/json")
    for name, value in headers.items():
        if value:
            req.add_header(name, value)
    with urllib.request.urlopen(req) as response:
        return json.loads(response.read())


def embed(texts):
    auth = {"Authorization": "Bearer " + TEI_KEY} if TEI_KEY else {}
    return call("POST", TEI + "/embed", {"inputs": texts}, auth)


qd = {"api-key": QDRANT_KEY}
docs = [
    "TEI serves embedding and reranker models over HTTP.",
    "Qdrant stores vectors and finds the nearest ones.",
    "Caddy gets HTTPS certificates automatically.",
]
vectors = embed(docs)
size = len(vectors[0])
print("vector size:", size)

call("PUT", QDRANT + "/collections/tei_demo", {"vectors": {"size": size, "distance": "Cosine"}}, qd)
points = [{"id": i, "vector": v, "payload": {"text": t}} for i, (v, t) in enumerate(zip(vectors, docs), 1)]
call("PUT", QDRANT + "/collections/tei_demo/points?wait=true", {"points": points}, qd)

query = embed(["Which tool stores vectors?"])[0]
found = call("POST", QDRANT + "/collections/tei_demo/points/query", {"query": query, "limit": 2, "with_payload": True}, qd)
for point in found["result"]["points"]:
    print(round(point["score"], 3), point["payload"]["text"])

Set TEI_API_KEY (and QDRANT_API_KEY if Qdrant has a key) and run it with python3 (Linux, Mac) or python (Windows). The paths follow Qdrant's API file for v1.20.0; we did not run it against live servers.

Reranking: a format mismatch#

TEI's /rerank takes query and texts and returns [{index, score}]. Open WebUI's external reranker and LightRAG's cohere and jina bindings send documents and expect results[].relevance_score. By reading their code, TEI would answer those calls with a validation error. We know of no setting that fixes this; leave reranking off in those two tools or put a small adapter in between (we did not build one). RAGFlow's HuggingFace rerank provider already speaks TEI's format.

Put it behind HTTPS (with or without Traefik)#

Containers on the same Docker network talk plain HTTP to http://tei:80 and need nothing here. Use HTTPS when another machine calls TEI, because the API key travels in a header. Both routes need a DNS record for tei.<your-domain> and ports 80 and 443 free. Otherwise skip this section.

With Traefik#

On a server that already runs Traefik v3 with the Docker provider, save this as docker-compose.traefik.yml next to docker-compose.yml. It assumes an entrypoint called https; use yours:

services:
  tei:
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.tei.rule=Host(`tei.<your-domain>`)"
      - "traefik.http.routers.tei.entrypoints=https"
      - "traefik.http.routers.tei.tls=true"
      - "traefik.http.services.tei.loadbalancer.server.port=80"

docker compose -f docker-compose.yml -f docker-compose.traefik.yml up -d

Check it worked. curl -I https://tei.<your-domain>/health returns HTTP/2 200 with a valid certificate.

Without Traefik: Caddy#

Choose this if you have no Traefik. Caddy gets and renews certificates by itself. Our Caddy guide goes deeper.

Create the folders, with a conf folder (Caddy's Docker docs warn against mounting a single Caddyfile):

mkdir -p ~/docker_volumes/caddy/{conf,data,config}

Save ~/docker_volumes/caddy/docker-compose.yml. The image is caddy:2.11.8, the latest release when we checked in November 2026:

services:
  caddy:
    image: caddy:2.11.8
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80/tcp"
      - "443:443/tcp"
      - "443:443/udp"
    volumes:
      - ./conf:/etc/caddy
      - ./data:/data
      - ./config:/config
    networks:
      - proxy
networks:
  proxy:
    external: true

Save ~/docker_volumes/caddy/conf/Caddyfile for a public domain:

tei.example.com {
    reverse_proxy tei:80
}

Or, for a home-only name (point it at the server in your router or hosts file):

tei.home.arpa {
    tls internal
    reverse_proxy tei:80
}

Start Caddy: cd ~/docker_volumes/caddy && docker compose up -d. For a home-only certificate, copy Caddy's root out and trust it on each device (the OS steps are ours, not from the Caddy docs):

docker cp caddy:/data/caddy/pki/authorities/local/root.crt ./caddy-root.crt

• Linux: copy it to /usr/local/share/ca-certificates/caddy-root.crt and run sudo update-ca-certificates.
• Windows: Import-Certificate -FilePath .\caddy-root.crt -CertStoreLocation Cert:\CurrentUser\Root.
• Mac: sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain caddy-root.crt.

Windows and Mac are clients here: run the Caddy commands on the Linux server.

Check it worked. curl -I https://tei.example.com/health returns a status line over HTTPS. For tls internal, the browser shows a padlock only after the root is trusted on that device.

Tip: Let's Encrypt needs the DNS record to point at the server and ports 80 and 443 to reach it. A certificate error usually means the name in the Caddyfile differs from the DNS record.

Security and testing#

What leaves the machine: the model download from huggingface.co on first start (and any later download of a new model), and nothing else we found. The docs describe no telemetry; tracing goes out only if you set --otlp-endpoint.

Quick tests: docker compose ps, docker compose logs --tail=100 tei, and the /health call from Step 3 (on Windows, the Invoke-WebRequest line).

Common mistakes#

These come from the official docs and the issue tracker at github.com/huggingface/text-embeddings-inference (read November 2026), and from common AI assistant answers.

Tip: After every start, call /info and read model_id and max_input_length. It proves which model is loaded and tells you the longest text it embeds in full.

Tip: Write the model name, its dimension count and the TEI tag in a note the day you start. They are the three settings you cannot change later without re-indexing.

Tip: Test with one short sentence before you index a thousand files. A wrong model, key or URL shows up in one call instead of after an hour.

Troubleshooting#

Known bugs#

• Issue #837 (open): slow start with the generic CUDA image. Use the compute-capability image.
• Issue #760 (open): Qwen3-Embedding-0.6B ran out of memory on a Turing image with the default --max-batch-tokens (CUDA_ERROR_INVALID_VALUE). Lower it.

Diagnostics: docker compose ps, docker compose logs --tail=200 tei, docker inspect tei, docker stats tei.

Updates and backups#

Update. Read the release notes, change TEI_TAG in .env, then:

cd ~/docker_volumes/tei
docker compose pull
docker compose up -d
docker compose logs --tail=100 tei

Keep the same model, or index again. On Windows run the same commands in $HOME\docker_volumes\tei.

Back up. The model cache in data can be downloaded again, so the important files are docker-compose.yml and .env (it holds your key). Archive them, with the cache if you want an offline copy:

cd ~/docker_volumes/tei
tar -czf "backups/tei-$(date +%F).tar.gz" docker-compose.yml .env data
chmod 600 backups/tei-*.tar.gz

In PowerShell:

Set-Location "$HOME\docker_volumes\tei"
Compress-Archive -Path docker-compose.yml, .env, data -DestinationPath "backups\tei-$(Get-Date -Format yyyy-MM-dd).zip"

Schedule it: on Linux and Mac save the commands as backup.sh and add 0 3 * * * ~/docker_volumes/tei/backup.sh with crontab -e. On Windows save the block as backup.ps1 and run schtasks /Create /SC DAILY /ST 03:00 /TN "TEI backup" /TR "powershell -NoProfile -File $HOME\docker_volumes\tei\backup.ps1". The archive holds your key, so keep it private; our restic guide shows an off-machine copy.

Restore test. Unpack into a new empty folder, never over the live one:

mkdir -p /tmp/restore-test && tar -xzf backups/tei-<date>.tar.gz -C /tmp/restore-test
ls /tmp/restore-test

On Windows use Expand-Archive into a temporary folder. We did not run these commands.

Performance and resources#

The TEI docs publish no RAM, CPU or speed figures for the Docker setup, so this guide gives none of its own. RAGFlow's .env file comments that bge-small-en-v1.5 "requires 1.2GB RAM/vRAM to load" and Qwen3-Embedding-0.6B 25 GB; those are RAGFlow's notes for its setup, not TEI numbers. Measure your own:

• Measure: docker stats tei shows live CPU and memory.
• --max-batch-tokens: the docs call it "one critical control" for using the hardware. Default 16384; TEI cannot work out the best value itself. Issue #760 shows too high a value can crash a Turing GPU.
• Timeouts in clients: RAGFlow waits 30 seconds per call, and LightRAG's default EMBEDDING_TIMEOUT is 30. Big CPU batches can exceed that.

FAQ#

Is TEI free, and is it maintained? Yes to both: Apache-2.0, v1.9.4 in September 2026, a commit in November 2026. Models have their own licences.

Do I need a GPU? No. The CPU image works; a GPU speeds up large models and big batches.

Can I use TEI instead of OpenAI's embedding API? For embeddings, yes: /v1/embeddings follows the OpenAI format. Point the tool's base URL at TEI and use the TEI key.

Which model should I start with? BAAI/bge-small-en-v1.5 for English on a CPU; Qwen/Qwen3-Embedding-0.6B for other languages or long input.

Can I swap the model later? Not cheaply. Create a new collection and index again.

Sources#

• TEI release v1.9.4, README, LICENSE, Dockerfile and openapi.json - read November 2026
• TEI docs at v1.9.4: Quick Tour, Supported models and hardware, and Metal - read November 2026
• TEI image on GitHub Container Registry (tags cpu-1.9.4, cpu-arm64-1.9.4 and the GPU tags checked) - November 2026
• Hugging Face brand assets and the model pages for bge-small-en-v1.5, Qwen3-Embedding-0.6B, bge-reranker-base, multilingual-e5-large-instruct and embeddinggemma-300m - November 2026
• Open WebUI environment variable docs and its reranker source; LightRAG env.example v1.5.7; RAGFlow v0.27.3 .env, compose and model code; Qdrant API v1.20.0 - November 2026
• Caddy release v2.11.8, Caddy reverse_proxy, Caddy tls and running Caddy with Docker - November 2026
• TEI GitHub issues #50, #53, #232, #268, #644, #658, #729, #760, #813 and #837 - read November 2026

Checked against the official docs on 2026-11-05.