AnythingLLM + Ollama with Docker: Chat with Your Own Documents (RAG) on Linux, Windows and Mac

What you get in one minute#

AnythingLLM is an open-source chat app that answers questions from your own files, and Ollama runs the AI models on your own computer, so nothing leaves your machine.

You upload PDFs, Word files or text, AnythingLLM cuts them into pieces, and when you ask a question it finds the best pieces and hands them to the chat model. This is RAG (retrieval-augmented generation). The model does not learn your documents; it reads the matching pieces again for every question. The answer comes with citations, so you can see where it came from. AnythingLLM runs in one Docker container and has a vector database (LanceDB) built in, so you need no extra database container. Ollama runs directly on your computer, which is also the way to use the GPU on a Mac with Apple Silicon.

How it fits together#

Your documents are split into chunks and turned into vectors by an embedding model in Ollama, AnythingLLM stores them in LanceDB, and each question pulls back the closest chunks for the chat model to answer from.

The container reaches Ollama on the host through the name host.docker.internal. Inside the container, localhost means the container itself, so never use it for Ollama.

Before you start#

You need Docker, a computer with enough memory for a small model, and about 15 GB of free disk space.

Check Docker first:

docker --version
docker compose version

Checklist:

• [ ] Docker runs (docker ps answers without an error)
• [ ] Port 3001 is free on your computer
• [ ] About 15 GB of free disk space for Ollama, two models and the AnythingLLM image
• [ ] Any older AnythingLLM data you want to keep is backed up

Not run by DIY Lab Hub: every command below was checked against the official documentation on 2026-10-26, not run on a test machine. Where the docs leave something open, the text says so.

Step 1: Install Ollama on your computer#

Install Ollama as a normal program on the host, not in Docker, so it can use your GPU directly.

Linux#

curl -fsSL https://ollama.com/install.sh | sh

To pin the version from this guide, run curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.40.11 sh instead. Then check:

sudo systemctl status ollama
ollama --version

Windows (PowerShell)#

Download OllamaSetup.exe from ollama.com/download and run it. It installs per user and needs no administrator rights. Then open a new PowerShell window:

ollama --version

Mac (Terminal)#

Download the app from ollama.com/download, drag it to Applications and open it once. Then in Terminal:

ollama --version

Check it worked: ollama --version prints a version number. The documented Windows installer path needs at least 4 GB of free disk for Ollama itself, before any models.

Potential issue: if the Linux service is not active, sudo systemctl status ollama shows why; journalctl -u ollama --no-pager has the log.

Step 2: Pull a chat model and an embedding model#

You need two models: one that writes answers and a separate one that turns text into vectors.

The commands are the same on every system. gemma4:e2b is the example model from the Ollama quickstart (about 7.2 GB download). Pick a smaller or larger model if you like; just use the exact tag you pulled later.

Linux#

ollama pull gemma4:e2b
ollama pull nomic-embed-text
ollama list
ollama run gemma4:e2b "Reply with one short sentence confirming you are running."

Windows (PowerShell)#

ollama pull gemma4:e2b
ollama pull nomic-embed-text
ollama list
ollama run gemma4:e2b "Reply with one short sentence confirming you are running."

Mac (Terminal)#

Same commands as Linux:

ollama pull gemma4:e2b
ollama pull nomic-embed-text
ollama list
ollama run gemma4:e2b "Reply with one short sentence confirming you are running."

Check it worked: ollama list shows both names, and the model answers with a short sentence. The wording of the answer differs every time.

Never use the chat model as the embedding model. They do different jobs, and the embedding model must stay the same for indexing and for questions.

Step 3: Let the container reach Ollama#

Ollama listens only on 127.0.0.1 by default, so tell it to listen on all interfaces and keep it away from the internet.

Docker containers connect from another network address, so a localhost-only Ollama is invisible to them. Set OLLAMA_HOST as the Ollama FAQ describes.

Linux#

sudo systemctl edit ollama.service

In the editor that opens, add these two lines, save, and exit:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Then restart the service:

sudo systemctl daemon-reload
sudo systemctl restart ollama

Windows (PowerShell)#

Quit Ollama from the tray icon first. Then:

[Environment]::SetEnvironmentVariable("OLLAMA_HOST","0.0.0.0:11434","User")

Start Ollama again from the Start menu.

Mac (Terminal)#

launchctl setenv OLLAMA_HOST "0.0.0.0:11434"

Quit Ollama from the menu bar and open it again.

Security note: 0.0.0.0 means all network interfaces, and the Ollama API has no login. Anyone who can reach port 11434 can use your models. Never forward this port on your router, and on Linux restrict it with a firewall rule to your local and Docker networks (for example sudo ufw allow from <docker-subnet> to any port 11434, and no rule for anyone else). The Docker subnet is shown by docker network inspect bridge.

Step 4: Start AnythingLLM in Docker#

Run one container with port 3001 bound to your own computer and all data in ~/docker_volumes/anythingllm.

The official image keeps its data in /app/server/storage and reads settings from a mounted /app/server/.env. --cap-add SYS_ADMIN is needed for the sandboxed browser that fetches web pages. The tag is pinned to 1.18.0; the official docs show latest, so move the tag on purpose when you upgrade.

Linux#

export STORAGE_LOCATION="$HOME/docker_volumes/anythingllm"
mkdir -p "$STORAGE_LOCATION" && touch "$STORAGE_LOCATION/.env"
sudo chown -R 1000:1000 "$STORAGE_LOCATION"
docker run -d --name anythingllm --restart unless-stopped \
  -p 127.0.0.1:3001:3001 --cap-add SYS_ADMIN \
  --add-host=host.docker.internal:host-gateway \
  -v "${STORAGE_LOCATION}:/app/server/storage" \
  -v "${STORAGE_LOCATION}/.env:/app/server/.env" \
  -e STORAGE_DIR="/app/server/storage" \
  mintplexlabs/anythingllm:1.18.0

The image runs as user and group 1000, so the folder must belong to it (the chown line). Docker Engine on Linux does not provide host.docker.internal on its own; --add-host adds it. The AnythingLLM docs also show 172.17.0.1 as a Linux alternative.

Windows (PowerShell)#

$StorageLocation = Join-Path $HOME "docker_volumes\anythingllm"
$EnvFile = Join-Path $StorageLocation ".env"
New-Item -ItemType Directory -Force $StorageLocation | Out-Null
if (-not (Test-Path $EnvFile)) { New-Item -ItemType File $EnvFile | Out-Null }
docker run -d --name anythingllm --restart unless-stopped `
  -p 127.0.0.1:3001:3001 --cap-add SYS_ADMIN `
  --mount "type=bind,source=$StorageLocation,target=/app/server/storage" `
  --mount "type=bind,source=$EnvFile,target=/app/server/.env" `
  -e STORAGE_DIR="/app/server/storage" `
  mintplexlabs/anythingllm:1.18.0

Docker Desktop provides host.docker.internal itself, so no extra flag is needed.

Mac (Terminal)#

Same commands as Linux, without the chown line and without --add-host (Docker Desktop provides the name):

export STORAGE_LOCATION="$HOME/docker_volumes/anythingllm"
mkdir -p "$STORAGE_LOCATION" && touch "$STORAGE_LOCATION/.env"
docker run -d --name anythingllm --restart unless-stopped \
  -p 127.0.0.1:3001:3001 --cap-add SYS_ADMIN \
  -v "${STORAGE_LOCATION}:/app/server/storage" \
  -v "${STORAGE_LOCATION}/.env:/app/server/.env" \
  -e STORAGE_DIR="/app/server/storage" \
  mintplexlabs/anythingllm:1.18.0

Check it worked: open http://localhost:3001 in your browser. The AnythingLLM setup screen appears. Also read the log:

docker logs --tail 100 anythingllm

To prove the container can see Ollama, call Ollama from inside the container (the image has Node.js, so no curl is needed). This one-liner is our own example, not from the official docs:

docker exec anythingllm node -e 'fetch("http://host.docker.internal:11434/api/tags").then(async r=>{console.log(r.status);console.log(await r.text())}).catch(e=>{console.error(e);process.exit(1)})'

A status of 200 followed by a JSON list of your models means the link works. In Windows PowerShell the quoting of this one-liner is awkward, so run it in a WSL shell, or skip it and use the connection test in the first run below.

Potential issue: unable to open database file: ../storage/anythingllm.db in the log means the storage folder is not writable for user 1000. Fix the owner (Linux) instead of using chmod 777.

First run (same on all systems): connect AnythingLLM to Ollama and create a workspace#

In the setup screen choose Ollama for the chat model and a separate Ollama embedder, keep LanceDB, and create a workspace.

Follow the screens in this order:

1. LLM provider: Ollama. Base URL: http://host.docker.internal:11434. Model: gemma4:e2b (or the tag you pulled).
2. Embedder: Ollama again, with the same base URL, and the model nomic-embed-text. Do not choose the chat model here.
3. Vector database: LanceDB (the default). No extra container is needed.
4. Create a workspace, for example "My documents".

If the model list stays empty, the base URL or the Ollama binding is wrong; see Troubleshooting.

Check it worked: the workspace opens and you can type a message. If you only ask something general, the chat model answers without any documents. That already proves the Ollama link.

Add documents and ask questions#

Upload a file, move it into the workspace and embed it, then ask a question you can check yourself.

1. In the workspace, click the upload icon, add a PDF, Word or text file.
2. Select the file, move it into the workspace and click the embed button. Uploading and embedding are two steps; until you embed, the file is not searchable.
3. Open the workspace settings and set the chat mode to Query for a first test.
4. Ask a question whose answer you know is in the file, using words from the document.

The chat modes differ:

Check it worked: the answer is followed by citations that list the document pieces used. Open one and compare it with your file. A citation shows which text was retrieved; it does not prove the model read it correctly, so check important answers yourself.

Embedded files are available to all threads of that workspace and to every user with access to it.

Run it permanently with Docker Compose#

A Compose file keeps the same settings in one place, which makes updates and moves easy.

First stop the container from Step 4, because the name is taken: docker rm -f anythingllm. Your data stays in the storage folder. Create the Compose file in its own folder so it does not mix with the data:

services:
  anythingllm:
    image: mintplexlabs/anythingllm:1.18.0
    container_name: anythingllm
    restart: unless-stopped
    ports:
      - "127.0.0.1:3001:3001"
    cap_add:
      - SYS_ADMIN
    environment:
      STORAGE_DIR: /app/server/storage
    volumes:
      - ${STORAGE_LOCATION}:/app/server/storage
      - ${STORAGE_LOCATION}/.env:/app/server/.env
    extra_hosts:
      - "host.docker.internal:host-gateway"

Save it as ~/docker_volumes/anythingllm-compose/docker-compose.yml (Windows: $HOME\docker_volumes\anythingllm-compose\docker-compose.yml). The extra_hosts lines are for Linux; Docker Desktop on Windows and Mac already provides the name, so you can remove them there. Docker Compose reads STORAGE_LOCATION from your shell, so set it first.

Linux#

export STORAGE_LOCATION="$HOME/docker_volumes/anythingllm"
cd ~/docker_volumes/anythingllm-compose
docker compose config
docker compose up -d

Windows (PowerShell)#

$env:STORAGE_LOCATION = (Join-Path $HOME "docker_volumes/anythingllm").Replace("\","/")
Set-Location "$HOME\docker_volumes\anythingllm-compose"
docker compose config
docker compose up -d

Mac (Terminal)#

Same commands as Linux:

export STORAGE_LOCATION="$HOME/docker_volumes/anythingllm"
cd ~/docker_volumes/anythingllm-compose
docker compose config
docker compose up -d

The .env that AnythingLLM reads (mounted into the container) is not the file Compose uses for variables. Do not confuse them. Check it worked: docker compose ps shows anythingllm as running and http://localhost:3001 opens with your workspace.

Put it behind HTTPS with Traefik#

This is only for a server with a domain; if you use AnythingLLM on one computer, skip this section.

It assumes Traefik already runs with the Docker provider, an entrypoint named https and a certificate for your hostname. Create the shared network once if it is missing (docker network create proxy), remove the published ports: from the Compose file and use:

services:
  anythingllm:
    image: mintplexlabs/anythingllm:1.18.0
    container_name: anythingllm
    restart: unless-stopped
    cap_add:
      - SYS_ADMIN
    environment:
      STORAGE_DIR: /app/server/storage
    volumes:
      - ${STORAGE_LOCATION}:/app/server/storage
      - ${STORAGE_LOCATION}/.env:/app/server/.env
    extra_hosts:
      - "host.docker.internal:host-gateway"
    networks:
      - default
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.anythingllm.rule=Host(`llm.<your-domain>`)"
      - "traefik.http.routers.anythingllm.entrypoints=https"
      - "traefik.http.routers.anythingllm.tls=true"
      - "traefik.http.services.anythingllm.loadbalancer.server.port=3001"
networks:
  proxy:
    external: true

Add traefik.http.routers.anythingllm.tls.certresolver=<your-resolver> if you use an ACME resolver. The official guidance for public setups is a TLS reverse proxy that also passes WebSocket connections, which the agent feature uses. After the change, test that streaming answers, file uploads and agent mode still work.

Check it worked: curl -I https://llm.<your-domain>/ answers with a normal HTTP status.

Security and testing#

AnythingLLM and Ollama have no strong protection by default, so who can reach them is your job.

Keep port 3001 off the internet; use the HTTPS section if people outside your network must reach it.

Quick tests after every change:

docker compose ps
docker logs --tail 100 anythingllm

Then open the page, ask a question about an embedded document and confirm the citations appear.

Using it for RAG: what makes answers good#

Better answers come from clean documents, questions that use the document's words and a similarity setting that is not too strict.

The default chunk size is 1,000 characters (characters, not tokens) with an overlap of 20. The setting only applies to files you embed afterwards, so re-embed a file to test a new value. The same embedding model must be used for indexing and for questions; if you change it, embed everything again. If the answer says nothing relevant was found, check that the file is embedded in this workspace, set the similarity threshold to "No restriction" and rephrase with words that appear in the document. Scanned PDFs without text have nothing to extract.

Troubleshooting#

Diagnostics:

docker logs --tail 200 anythingllm
docker compose ps
docker inspect anythingllm
journalctl -u ollama --no-pager --follow --pager-end

The journalctl line is for Linux. On Mac, read tail -n 200 ~/.ollama/logs/server.log. On Windows, run Get-Content "$env:LOCALAPPDATA\Ollama\server.log" -Tail 200 in PowerShell.

Cannot reach Ollama from the container#

The most common cause is localhost in the base URL. The container's own localhost is the container. The second cause is Ollama listening only on 127.0.0.1; set OLLAMA_HOST as in Step 3 and restart Ollama. Then repeat the docker exec ... node -e test from Step 4.

unable to open database file#

The exact line in the log is unable to open database file: ../storage/anythingllm.db. The container user (1000) cannot write to the folder you mounted. Change the owner as shown in the table; do not open the folder to everyone.

Illegal instruction (core dumped)#

LanceDB needs a CPU with AVX2 on x86. Older CPUs and some virtual machine CPU types hide it. ARM64 computers are not affected.

Updates and backups#

Back up the whole storage folder (with the hidden .env) before every update, and test the restore once.

Ollama models are stored separately. You only need to back them up to avoid downloading them again.

Linux#

mkdir -p ~/docker_volumes/anythingllm-backups
docker stop anythingllm
tar -czf ~/docker_volumes/anythingllm-backups/anythingllm-$(date +%F).tar.gz -C ~/docker_volumes anythingllm
docker start anythingllm

Windows (PowerShell)#

New-Item -ItemType Directory -Force "$HOME\docker_volumes\anythingllm-backups" | Out-Null
docker stop anythingllm
Compress-Archive -Path "$HOME\docker_volumes\anythingllm" -DestinationPath "$HOME\docker_volumes\anythingllm-backups\anythingllm-$(Get-Date -Format yyyy-MM-dd).zip" -Force
docker start anythingllm

Mac (Terminal)#

Same commands as Linux:

mkdir -p ~/docker_volumes/anythingllm-backups
docker stop anythingllm
tar -czf ~/docker_volumes/anythingllm-backups/anythingllm-$(date +%F).tar.gz -C ~/docker_volumes anythingllm
docker start anythingllm

Stopping the container first gives a consistent copy. Put the same commands into a script and run it every day at 03:00: with cron on Linux and Mac (0 3 * * *), or with Task Scheduler on Windows. Keep a copy on another machine, and keep your Compose file and the image tag with it.

Restore test: extract the archive into a temporary folder and check that it holds your workspace data and the .env file. To restore for real, stop the container, put the folder back and start it with the same image tag.

Updating AnythingLLM: back up, change the tag in the Compose file, then:

docker compose pull
docker compose up -d

Open the page and check login, workspaces and one question with citations. Going back to an older version after a newer one has changed the database is not guaranteed to work; restore the backup together with the old tag instead.

Updating Ollama: on Linux run the install script again; on Windows and Mac install the new release from ollama.com.

Performance and resources#

Memory use depends mostly on the chat model and its context length, not on AnythingLLM.

• Watch the container with docker stats and the models with ollama ps.
• A larger context uses more memory. Set OLLAMA_CONTEXT_LENGTH=8192 the same way you set OLLAMA_HOST (Step 3) and see if it fits.
• Ollama keeps a model loaded for 5 minutes after the last use by default. OLLAMA_KEEP_ALIVE changes that.
• If the model does not fit in GPU or unified memory, Ollama falls back to normal RAM, which is slower.
• Smaller chunks find precise facts; larger chunks keep more context. Change one value at a time and re-embed.
• The container itself needs no GPU, because Ollama does the inference on the host.

We did not measure RAM or speed for this guide; use docker stats on your own machine for real numbers.

FAQ#

Does uploading a file train the model? No. The text is split, turned into vectors and stored. The model reads the matching pieces for each question and learns nothing.

Does it work offline? Yes, once the images and models are downloaded, as long as you use Ollama and LanceDB and do not switch on web features that fetch pages.

Does the container need GPU access? No. Ollama on the host runs the models. AnythingLLM only needs to reach it over the network.

Why does the answer ignore my document? Usually the file is uploaded but not embedded in this workspace, the similarity threshold is too strict, or the question does not use the document's words.

Can several people use it safely? Yes, with multi-user mode. Give each person access only to their workspaces, because a workspace is the permission boundary.

Sources#

• AnythingLLM releases (v1.18.0, 13 October 2026) and Docker Hub tags
• AnythingLLM docs: local Docker, system requirements, localhost access, Ollama connection troubleshooting
• Ollama embedder, LanceDB, text splitting, chat modes, security and access
• Ollama releases (v0.40.11, 24 October 2026); Ollama docs for Linux, Windows, macOS, FAQ and troubleshooting
• Most of these documentation pages carry no date; they were read on 2026-10-26.
• Checked against the official docs on 2026-10-26 (not run on a test machine by DIY Lab Hub).