Which Local LLM Fits Your GPU? A VRAM Guide for Ollama and llama.cpp (Linux, Windows, Mac)

What you get in one minute#

A local LLM runs fast only when the model file, the context memory and a little overhead all fit in your GPU memory (VRAM). This guide gives you the official download sizes, a simple way to check the fit, and the commands to run it with Ollama or llama.cpp.

At a glance:

• You get: a size table of current open models (Qwen, Gemma, Llama, Mistral, DeepSeek, gpt-oss) and a pick list for 8, 12, 16, 24 and 32 GB, plus Apple Silicon.
• Time needed: 10 minutes to read, about 20 minutes to set up Ollama and test a model (plus the model download).
• Difficulty: easy. You copy commands and read one status table.
• You need: a PC with an NVIDIA GPU (or an Apple Silicon Mac), 20 GB or more free disk, and Docker only if you want the container route.

Why the latest Ollama? It ships almost daily, so we read the issue tracker for the last two weeks. Open reports are narrow (an AMD iGPU, one chat error, Mac Metal crashes). No wave of install failures or lost data, so we use the current release. AMD iGPU owners: see Known bugs.

How it fits together#

Three things share your VRAM: the model weights (the file you download), the context memory (the KV cache, which grows with context length), and a little overhead. If the sum is too big, the rest spills into slow system RAM.

Before you start#

You need a supported GPU and a current driver. Docker is optional.

AMD Radeon cards work through ROCm or Vulkan; see the Ollama hardware page.

Check your hardware first.

Linux#

nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv
free -h
df -h ~

Windows (PowerShell)#

nvidia-smi --query-gpu=name,memory.total,memory.used --format=csv
Get-CimInstance Win32_ComputerSystem | Select-Object TotalPhysicalMemory
Get-PSDrive C

Mac (Terminal)#

sysctl -n hw.memsize | awk '{printf "%.0f GB unified memory\n", $1/1024/1024/1024}'
system_profiler SPDisplaysDataType | grep -E "Chipset Model|Total Number of Cores"

Check list:

• [ ] I know my VRAM (or my Mac's memory) in GB
• [ ] Port 11434 is free
• [ ] At least 20 GB free disk
• [ ] Only for the Traefik section: a domain, DNS and ports 80/443

Tested on: not run by DIY Lab Hub for this guide (Linux = not run, Windows = not run, Mac = not run). Commands and sizes were checked against the official Ollama and llama.cpp pages on 2026-10-26 (see Sources).

How your VRAM is used#

Weights are the big, fixed part. The KV cache is the part you can control. Overhead is small but real.

Quantization in plain words#

Quantization stores each weight with fewer bits: a smaller file, a small loss in quality.

Rule of thumb from DIY Lab Hub (not official): start with q4_K_M; go up to q8_0 only with spare VRAM.

Context length and the KV cache#

Ollama picks a default context length from your VRAM. Per the official context length page:

The same page says agents and coding tools should use at least 64000 tokens. A longer context needs more memory. Ollama's own numbers for the KV cache:

• OLLAMA_KV_CACHE_TYPE=f16 is the default.
• q8_0 uses about half the memory of f16, usually with no noticeable quality loss.
• q4_0 uses about a quarter, with a small to medium loss.
• It needs Flash Attention, which Ollama turns on by itself when your hardware supports it.

The size table: current open models#

These are the official download sizes in GB at the quantization the Ollama library lists. A dash means that tag does not exist or we did not need it.

Read on 2026-10-26 from the tag pages on ollama.com. Sizes are the file you download, not the VRAM you need.

Notes on the table:

• MoE models (a3b, a4b) use few parameters per token, but the whole file must sit in memory. The file size is what counts.

Which model for which memory?#

Pick the biggest model whose file leaves headroom for the context memory.

DIY Lab Hub rule of thumb (not official): keep the file at about three quarters of your VRAM or less. Then raise the context step by step and watch ollama ps (see below). The sizes are the official ones from the table.

Apple Silicon (unified memory)#

On a Mac the CPU and GPU share one pool of memory. The file sizes in the table are the same. The difference: macOS keeps part of the pool for itself and does not give all of it to the GPU. Neither Ollama nor Apple publishes the exact share, and it differs by Mac and macOS version, so keep a wide margin and trust ollama ps.

CPU offload#

When a model does not fit, Ollama puts some layers in system RAM and runs them on the CPU. It works but is slower (we did not measure by how much). ollama ps shows the split (Step 3).

Step 1: Install Ollama#

For a single PC you use yourself, the native app is the simplest way and gives the best GPU support.

Linux#

curl -fsSL https://ollama.com/install.sh | sh
ollama --version

To pin this guide's version, use the variable from the official Linux page:

curl -fsSL https://ollama.com/install.sh | OLLAMA_VERSION=0.40.11 sh

Windows (PowerShell)#

Download OllamaSetup.exe from ollama.com/download and run it. It installs in your home folder and needs no administrator rights. Then open a new PowerShell:

ollama --version

Mac (Terminal)#

Download ollama.dmg from ollama.com/download, open it and drag Ollama to Applications. Start it once. Then:

ollama --version

Check it worked: the command prints a version number (this guide used 0.40.11).

Step 2: Download a model that fits#

Pull a model from the "Good fits" column of your row. Start small, then go bigger.

Linux#

ollama pull qwen3:8b
ollama run qwen3:8b "Say hello in one short sentence."

Windows (PowerShell)#

ollama pull qwen3:8b
ollama run qwen3:8b "Say hello in one short sentence."

Mac (Terminal)#

Same commands as Linux:

ollama pull qwen3:8b
ollama run qwen3:8b "Say hello in one short sentence."

To choose a quantization, use the full tag from the library, for example ollama pull qwen3:8b-q8_0.

Potential problem: downloads can be tens of GB. Check free disk first.

Step 3: Check what really fits#

ollama ps and nvidia-smi show the truth. Trust them more than any table.

Linux#

ollama ps
nvidia-smi
watch -n 2 nvidia-smi --query-gpu=memory.used,memory.total --format=csv

Windows (PowerShell)#

ollama ps
nvidia-smi
nvidia-smi --query-gpu=memory.used,memory.total --format=csv -l 2

Mac (Terminal)#

ollama ps

For the whole Mac, see Activity Monitor, Memory tab.

How to read ollama ps. The official docs show this format (the values here are the docs' example, not ours):

NAME             ID              SIZE      PROCESSOR    CONTEXT    UNTIL
gemma4:latest    c6eb396dbd59    9.6 GB    100% GPU     131072     2 minutes from now

SIZE is the real memory use, including context. It is bigger than the file size.

Step 4: Set the context length#

Raise the context in small steps and watch ollama ps after each step.

Linux#

For a service started by systemd, set variables with sudo systemctl edit ollama, add them under [Service], then restart:

[Service]
Environment="OLLAMA_CONTEXT_LENGTH=16384"
Environment="OLLAMA_FLASH_ATTENTION=1"
Environment="OLLAMA_KV_CACHE_TYPE=q8_0"

sudo systemctl daemon-reload
sudo systemctl restart ollama

Windows (PowerShell)#

Quit Ollama from the tray. Set the variables for your user, then start Ollama again:

[Environment]::SetEnvironmentVariable("OLLAMA_CONTEXT_LENGTH","16384","User")
[Environment]::SetEnvironmentVariable("OLLAMA_FLASH_ATTENTION","1","User")
[Environment]::SetEnvironmentVariable("OLLAMA_KV_CACHE_TYPE","q8_0","User")

You can also move the slider in the Ollama app under Settings.

Mac (Terminal)#

launchctl setenv OLLAMA_CONTEXT_LENGTH 16384
launchctl setenv OLLAMA_FLASH_ATTENTION 1
launchctl setenv OLLAMA_KV_CACHE_TYPE q8_0

Restart the Ollama app afterwards.

Per request you can use /set parameter num_ctx 8192 in ollama run or "options": {"num_ctx": 8192} in an API call. Check it worked: ollama ps shows your number in the CONTEXT column.

Option: run Ollama in Docker#

Use Docker on a Linux server or a Windows PC with WSL2. On a Mac, use the native app.

You need the NVIDIA Container Toolkit first. On Ubuntu or Debian, the official Ollama Docker page gives:

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

On Windows, Docker Desktop with WSL2 already handles the GPU: skip the toolkit and use the Ubuntu shell (wsl -d Ubuntu). GPU details are in our vLLM guide.

Linux#

mkdir -p ~/docker_volumes/ollama/{data,config,logs,backups}
cd ~/docker_volumes/ollama

Create docker-compose.yml (there is no version: line):

services:
  ollama:
    image: ollama/ollama:0.40.11
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ./data:/root/.ollama
    environment:
      - OLLAMA_CONTEXT_LENGTH=16384
      - OLLAMA_FLASH_ATTENTION=1
      - OLLAMA_KV_CACHE_TYPE=q8_0
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

docker compose up -d
docker exec -it ollama ollama pull qwen3:8b
docker exec -it ollama ollama ps

Windows (PowerShell)#

Open the Ubuntu shell with wsl -d Ubuntu and use the Linux commands above, so your files live on the Linux side.

Mac (Terminal)#

Docker on a Mac cannot use the Apple GPU, so a container would run on CPU only. Use the native app from Step 1.

Check it worked: docker compose ps shows ollama as running, and docker exec -it ollama ollama ps lists your model with 100% GPU.

Potential problem: could not select device driver "" with capabilities: [[gpu]] means the toolkit is not set up. Repeat the nvidia-ctk line and restart Docker.

Option: llama.cpp for full control#

llama.cpp is the engine under many local tools. Use it when you want to choose the exact GGUF file and how many layers go to the GPU.

The official quick start installs it with one line and downloads a model straight from Hugging Face:

Linux#

curl -LsSf https://llama.app/install.sh | sh
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Windows (PowerShell)#

irm https://llama.app/install.ps1 | iex
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

Mac (Terminal)#

Same commands as Linux:

curl -LsSf https://llama.app/install.sh | sh
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

llama serve starts an OpenAI-compatible API with a built-in web page. That model is the README's small example. Swap in the GGUF repo of your pick; Hugging Face model cards list GGUF sizes per quantization.

For Docker on NVIDIA, the official images are server-cuda (CUDA 12) and server-cuda13. Pinned tags look like server-cuda-b<number>; server-cuda-b11889 exists today.

docker run --gpus all -v ~/docker_volumes/llamacpp/models:/models -p 127.0.0.1:8080:8080   ghcr.io/ggml-org/llama.cpp:server-cuda-b11889   -m /models/<your-model>.gguf --host 0.0.0.0 --port 8080

Put it behind HTTPS (with or without Traefik)#

With Traefik#

On a server with a domain, let Traefik publish Ollama over HTTPS and remove the host port. Local-only users can skip this.

Ollama has no login of its own, so never publish it without one. This assumes Traefik v3, an entrypoint https and a login middleware auth you already have. Create the network once: docker network create proxy. Delete the ports: block and merge:

services:
  ollama:
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.ollama.rule=Host(`llm.<your-domain>`)"
      - "traefik.http.routers.ollama.entrypoints=https"
      - "traefik.http.routers.ollama.tls=true"
      - "traefik.http.routers.ollama.middlewares=auth@docker"
      - "traefik.http.services.ollama.loadbalancer.server.port=11434"
networks:
  proxy:
    external: true

Add traefik.http.routers.ollama.tls.certresolver=<your-resolver> if you use ACME. Check it worked: curl -I https://llm.<your-domain>/api/tags answers 401 (or a redirect to login) without credentials.

Without Traefik: Caddy#

Choose Caddy when you have no Traefik yet and run one small server. Facts are from caddyserver.com/docs.

Public domain, Caddyfile for Ollama:

app.example.com {
    basic_auth {
        you <hash from: caddy hash-password>
    }
    reverse_proxy localhost:11434
}

Home only:

app.home.arpa {
    tls internal
    basic_auth {
        you <hash from: caddy hash-password>
    }
    reverse_proxy localhost:11434
}

Trust the root certificate (home only): on the Caddy machine run sudo caddy trust (Linux). On Windows and Mac copy pki/authorities/local/root.crt from Caddy's data folder and import it as a trusted root (Windows certificate manager, Mac Keychain Access).

Run Caddy (2.11.7 was current on 2026-10-26): Linux with the official apt repository from the install page, then sudo systemctl reload caddy; Windows winget install CaddyServer.Caddy then caddy run; Mac brew install caddy then caddy run. Or use Docker: put caddy:2.11.7 on the same network as the app, drop the app's ports: line and write reverse_proxy ollama:11434.

services:
  caddy:
    image: caddy:2.11.7
    container_name: caddy
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ~/docker_volumes/caddy/data:/data
      - ~/docker_volumes/caddy/config:/config

WebSockets work without extra settings. Ollama has no login of its own, so the examples add Caddy basic_auth; create the hash with caddy hash-password.

Check it worked: curl -I https://app.example.com answers HTTP/2 200 (or the login redirect). On Windows use curl.exe.

Security and testing#

Keep the port on your machine, keep secrets out of the Compose file, and test.

• Ollama binds 127.0.0.1:11434 by default. Do not change OLLAMA_HOST to 0.0.0.0 on a machine that others can reach, unless a login sits in front.
• You need no Ollama account locally. OLLAMA_NO_CLOUD=1 switches the cloud features off (official FAQ).
• Keep keys in a .env file with chmod 600, never in docker-compose.yml.

Quick tests:

docker compose ps
curl -s http://127.0.0.1:11434/api/tags

The second command lists the models you pulled.

Troubleshooting#

Server logs#

journalctl -u ollama --no-pager --follow --pager-end
docker logs ollama --tail 100

On Windows the logs are in %LOCALAPPDATA%\Ollama\server.log. On a Mac they are in ~/.ollama/logs/server.log. For more detail set OLLAMA_DEBUG=1.

Known bugs#

Open reports on the 0.40.x line (Ollama issue tracker, check before you blame your setup):

• AMD Radeon 890M iGPU on ROCm: garbage output and "Memory access fault by GPU" on 0.40.6 and 0.40.7; the reporter says 0.33.3 works (issue 19290).
• macOS Metal crashes (issue 19278).
• /api/chat sometimes fails with a 500 "broken pipe" right after a long reply (issue 19293).

Updates and backups#

Update the app or the image. Your models live in one folder and can be downloaded again, so back up the settings and the list of models.

Update#

Linux native: run the install script again. Windows and Mac: the app updates itself ("Restart to update"). Docker: change the tag in docker-compose.yml, then:

docker compose pull
docker compose up -d

Backup#

Models are large and come back with ollama pull. Back up your Compose file, .env and a list of your models:

cd ~/docker_volumes/ollama
docker exec ollama ollama list > config/models-$(date +%F).txt
tar czf backups/ollama-config-$(date +%F).tar.gz docker-compose.yml config

Schedule it: Linux and Mac with crontab -e (0 3 * * *), Windows with Task Scheduler running the same commands in WSL.

Restore test#

Test the restore in a throwaway folder, never over your live data:

mkdir -p /tmp/ollama-restore-test
tar xzf ~/docker_volumes/ollama/backups/ollama-config-$(date +%F).tar.gz -C /tmp/ollama-restore-test
ls /tmp/ollama-restore-test /tmp/ollama-restore-test/config
rm -rf /tmp/ollama-restore-test

You should see docker-compose.yml and your models list.

Performance and resources#

Fit first, then tune. Staying inside VRAM matters more than any setting.

• Keep the model on 100% GPU. A CPU split is the biggest slowdown you can avoid.
• Use the context you need, not the model maximum. Use OLLAMA_KV_CACHE_TYPE=q8_0 to halve cache memory.

Related: Open WebUI with Ollama, vLLM.

FAQ#

Is the file size the VRAM I need? No. File plus context memory plus overhead. ollama ps shows the real total.

Q4 or Q8: which should I pick? Start with q4_K_M. Take q8_0 only with spare VRAM.

How much memory does a Mac really give the model? Not all of it. macOS keeps a share. The official docs do not give the limit, so keep a margin and check ollama ps.

Sources#

All pages read on 2026-10-26.

• Ollama library tag pages: gpt-oss, qwen3, qwen3-coder, qwen3.5, qwen3.6, qwen3.8, gemma3, gemma4, llama3.1, llama3.2, llama3.3, mistral, mistral-small, phi4, deepseek-r1
• Ollama docs: context length, FAQ, Docker, Hardware support, Linux, Windows, macOS, Troubleshooting
• Ollama releases and Docker Hub image (tag 0.40.11 exists)
• Ollama issue tracker: issues 19290, 19278, 19293
• llama.cpp README, llama.cpp Docker docs, llama.cpp releases
• NVIDIA Container Toolkit install guide
• Caddy docs: automatic HTTPS, tls internal, reverse_proxy, install (read on 2026-10-26)

Checked against the official docs on 2026-10-26. Not run by DIY Lab Hub. The VRAM rules of thumb in this guide are ours, not official figures.