Unsloth with Docker: fine-tune a small LLM on your own GPU (Linux, Windows, Mac)

What you get in one minute#

Unsloth is an open-source tool that fine-tunes a small language model on your own GPU with much less memory than usual, and its official Docker image gives you a browser app (Unsloth Studio) to do it without writing code.

You bring a few hundred examples of the answers you want. Unsloth trains a small add-on (a LoRA adapter) on top of a 1-4B model. You compare the result with the original, export a GGUF file and run it in Ollama.

At a glance:

• You get: your own tuned small model, running locally, with no cloud and no per-token bill
• Time: about 30 minutes of your own time, plus the image pull and the model download
• Difficulty: medium. If you can copy commands and use a browser, you can do it
• You need: an NVIDIA GPU (the docs list 3.5 GB of VRAM as the minimum for a 3B model with QLoRA), Docker, a few hundred training examples
• Not sure it is the right tool? Read "Fine-tune or RAG?" below before you start

Which version, and is it stable? Every Unsloth release is tagged -beta, and the docs name no separate "stable" channel. We checked the issue tracker for the last four weeks. There are many small bug reports and feature requests, but none about lost data or a broken install. One real regression hit Windows Insider builds (every model load was killed by a false check). It was fixed on 25 October. It only affects the native Windows app, not the Linux container used here. So this guide uses the current release and pins the tag, so a surprise latest can never change your setup. Update on purpose, when you have read the release notes.

Fine-tune or RAG?#

Fine-tune to change how a model behaves. Use RAG to give it facts that change.

One note for fairness: Unsloth's own guide says fine-tuning can teach a model new knowledge too, not only style. That is true. It is just slower and more work to keep fresh than a search index. Many people use both.

How it fits together#

You open Studio in a browser, it trains on your GPU inside one container, and your files stay in folders on your disk.

Before you start#

You need an NVIDIA GPU, a current driver and Docker. A Mac takes a different route, shown in Step 1.

The docs give these minimum VRAM numbers for fine-tuning. They say real use can need more:

For a first run, stay at 1-4B and use QLoRA. It needs the least memory. The docs also say QLoRA with their dynamic 4-bit models has mostly caught up with LoRA in quality.

Check the tools:

docker --version
docker compose version
nvidia-smi

• [ ] nvidia-smi shows your GPU and driver 570.26 or newer
• [ ] Ports 8000 and 8888 are free
• [ ] About 30 GB free disk for the 10.7 GB image, models and exports (our estimate)
• [ ] Any old Unsloth folder is backed up
• [ ] Only for the Traefik section: a domain, DNS and ports 80/443

Tested on: not run by DIY Lab Hub for this guide (Linux = not run, Windows = not run, Mac = not run). Commands were checked against the official Unsloth docs on 2026-10-26 (see Sources).

Step 1: Let Docker see your GPU#

If docker run --gpus all ... nvidia-smi works, you are done with this step.

Linux#

Install Docker if you do not have it, then the NVIDIA Container Toolkit. Unsloth ships a helper script for the toolkit:

curl -fsSL https://get.docker.com -o get-docker.sh && sh get-docker.sh
curl -fsSL https://raw.githubusercontent.com/unslothai/unsloth/main/docker/install_nvidia_toolkit.sh -o install_nvidia_toolkit.sh
less install_nvidia_toolkit.sh
sudo -E bash install_nvidia_toolkit.sh

Read the script before you run it (the less line). It installs system packages. If Docker was already running, restart it afterwards: sudo systemctl restart docker.

Windows (PowerShell)#

1. Install the latest NVIDIA Windows driver.
2. Install Docker Desktop and update WSL:

wsl --update

3. In Docker Desktop, open Settings > General and check Use the WSL 2 based engine. Wait for the engine to start.

Mac (Terminal)#

Docker on a Mac cannot pass the Apple GPU into this image, so skip the container. The docs say Unsloth Studio runs natively on macOS, including chat, MLX models and training. The simplest way:

brew install --cask unsloth

That installs the Unsloth Desktop app. Prefer a browser app? Use the install script, then start Studio:

curl -fsSL https://unsloth.ai/install.sh | sh
unsloth studio -p 8888

Open http://127.0.0.1:8888, set a password, and follow Step 4 onward. The Studio screens are the same. We have not run Mac training ourselves, so treat speed as unknown.

Check it worked (Linux and Windows): this should print a table with your GPU name. Expected output is the normal nvidia-smi table, listing your own card.

docker run --rm --gpus all --entrypoint nvidia-smi unsloth/unsloth:v0.1.910-beta

On Windows, use the same command in PowerShell. This also pulls the big image, so it takes a while the first time.

Step 2: Create the folder, secrets and Compose file#

Everything lives in one folder: ~/docker_volumes/unsloth.

Linux#

mkdir -p ~/docker_volumes/unsloth/{data/work,data/huggingface,backups}
cd ~/docker_volumes/unsloth
umask 077
cat > .env << EOF
UNSLOTH_STUDIO_PASSWORD=$(openssl rand -hex 16)
JUPYTER_PASSWORD=$(openssl rand -hex 16)
EOF
chmod 600 .env

Windows (PowerShell)#

$d = "$HOME\docker_volumes\unsloth"
New-Item -ItemType Directory -Force "$d\data\work","$d\data\huggingface","$d\backups" | Out-Null
function New-Secret { $b = New-Object byte[] 16; [Security.Cryptography.RandomNumberGenerator]::Create().GetBytes($b); ($b | ForEach-Object { $_.ToString('x2') }) -join '' }
Set-Content -Path "$d\.env" -Value "UNSLOTH_STUDIO_PASSWORD=$(New-Secret)", "JUPYTER_PASSWORD=$(New-Secret)"
Set-Location $d

Mac (Terminal)#

Skip this step. You are using the native app.

Now the Compose file. It is the official docker run command from the docs, written as Compose and without the obsolete version: line. Create docker-compose.yml in the same folder (use any editor):

services:
  unsloth:
    image: unsloth/unsloth:v0.1.910-beta
    container_name: unsloth
    restart: unless-stopped
    ipc: host
    ulimits:
      memlock: -1
      stack: 67108864
    ports:
      - "127.0.0.1:8000:8000"
      - "127.0.0.1:8888:8888"
    env_file: .env
    volumes:
      - ./data/work:/workspace/host
      - ./data/huggingface:/workspace/.cache/huggingface
      - unsloth-studio:/opt/unsloth-studio
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
volumes:
  unsloth-studio:
    name: unsloth-studio

What the parts do:

Step 3: Start it and sign in#

Start the container, wait about a minute, and sign in as user unsloth.

Linux#

cd ~/docker_volumes/unsloth
docker compose up -d
docker compose logs -f unsloth

Windows (PowerShell)#

Set-Location "$HOME\docker_volumes\unsloth"
docker compose up -d
docker compose logs -f unsloth

Mac (Terminal)#

Open the Unsloth app, or http://127.0.0.1:8888 if you used the script.

Look for the line Unsloth container ready in the log. Press Ctrl+C to stop following; the container keeps running. Then:

1. Open http://localhost:8000.
2. Sign in as unsloth with the password from your .env file (UNSLOTH_STUDIO_PASSWORD).
3. If Studio asks you to choose a new password, do it now.

Heads-up from the docs: if you never change a generated password, Studio shuts itself down after an hour (UNSLOTH_STUDIO_BOOTSTRAP_TIMEOUT, default 3600 seconds). docker restart unsloth brings it back. Running Docker on another machine? localhost is your own computer. Use an SSH tunnel: ssh -L 8000:localhost:8000 user@your-server.

Check it worked: the Studio page loads and shows the chat screen.

docker compose ps

Step 4: Prepare a small dataset#

One JSON object per line, each with a messages list. Studio calls this the chatml format.

Studio accepts PDF, DOCX, JSONL, JSON, CSV and Parquet. For a first run, JSONL is the clearest. These three lines show the shape (the question and answers are made-up examples):

{"messages":[{"role":"user","content":"Reply in our house style: when is the next backup?"},{"role":"assistant","content":"Backups run nightly at 03:00. Check the backups folder for today's file."}]}
{"messages":[{"role":"user","content":"Reply in our house style: how do I restart the app?"},{"role":"assistant","content":"Run docker compose restart. Then open the page and sign in again."}]}
{"messages":[{"role":"user","content":"Reply in our house style: where are the logs?"},{"role":"assistant","content":"Run docker compose logs --tail 100. The newest lines are at the bottom."}]}

Save your file as my-data.jsonl. Where, per system:

Linux#

Save it in ~/docker_volumes/unsloth/data/work. In Studio you upload it from there in Step 5.

Windows (PowerShell)#

Save it in $HOME\docker_volumes\unsloth\data\work (plain UTF-8 text, one JSON object per line):

notepad "$HOME\docker_volumes\unsloth\data\work\my-data.jsonl"

Mac (Terminal)#

Same as Linux: save it in any folder you like, for example ~/unsloth-work/my-data.jsonl.

Three lines will not teach a model anything. Aim for at least a few hundred clean, consistent examples. Quality beats quantity, and the docs say the dataset decides how good the result is.

No data yet? Pick a public dataset from the Hugging Face search inside Studio for your first test. Studio also has "Data Recipes" that can turn PDFs and CSVs into a dataset.

Other formats Studio understands: alpaca (instruction / input / output columns) and sharegpt (conversations). auto guesses.

Step 5: Train a small LoRA#

Pick a 1-4B model, keep QLoRA and the defaults, and press Start.

The screens are the same on every system. Only the address differs.

Linux#

Studio is at http://localhost:8000.

Windows (PowerShell)#

Studio is at http://localhost:8000 (the same container, reached through Docker Desktop).

Mac (Terminal)#

Studio is in the Unsloth app, or at http://127.0.0.1:8888 if you used the script. The docs say MLX training works there. Pick a model that fits your unified memory.

Then, on every system:

1. In Studio, open Train, then Configure.
2. Choose model type Text, method QLoRA.
3. Search for a small instruct model, for example a 1B or 3B Llama or Qwen instruct model from the Unsloth list. Models whose names end in unsloth-bnb-4bit are the dynamic 4-bit versions the docs recommend.
4. Under Dataset, choose Local and upload my-data.jsonl. Set the format to chatml (or leave auto).
5. Open Parameters (Simple mode is fine).
6. Click Start Training.

The defaults in the docs, so you know what you are accepting:

Watch Current Run: loss, learning rate and GPU memory update live. The docs say a loss around 0.5 to 1.0 is often a good sign, but it depends on your data. If the loss drops to 0, you probably overfit. Fewer epochs help.

Out of memory? The docs say the usual cause is a batch size that is too high. Set it to 1 or 2 and raise gradient accumulation to keep the effective batch size.

Check it worked: the progress bar reaches 100%. Click Compare in Chat and ask the same question to the original and the tuned model. If the tuned one answers in your style, it worked.

Step 6: Export a GGUF and run it in Ollama#

Export in Studio, then give Ollama the GGUF file and a one-line Modelfile.

1. In Studio, open Export. Choose your training run and the last checkpoint.
2. Choose GGUF / llama.cpp as the export type. A q4_k_m quantization is the one the docs mark as recommended for a small, fast file.
3. Save locally (the export also offers pushing to Hugging Face; you do not need that).

The docs do not say exactly where the saved file lands for the Docker setup, so look in the download folder of your browser first and in the unsloth-studio volume second. (VERIFY on your system.) Put the file in a folder next to a text file called Modelfile with one line:

FROM ./my-tuned-q4_k_m.gguf

Linux#

cd ~/my-model
ollama create my-tuned -f Modelfile
ollama run my-tuned

Windows (PowerShell)#

Set-Location "$HOME\my-model"
ollama create my-tuned -f Modelfile
ollama run my-tuned

Mac (Terminal)#

Same commands as Linux.

Check it worked: ollama list shows my-tuned, and ollama run my-tuned answers in your style.

No Ollama yet? Our Ollama guide covers the install.

One warning from the Unsloth docs: if the model works well in Unsloth but gives gibberish or endless text in Ollama, the cause is usually a different chat template. Use the same template for training and for running. If this happens, add the right TEMPLATE to the Modelfile for your base model.

Put it behind HTTPS (with or without Traefik)#

With Traefik#

On a server with a domain, let Traefik publish Studio and keep JupyterLab private.

This is for a machine with a domain. Local-only users can skip it. Studio and JupyterLab speak plain HTTP, and the docs warn not to expose them directly. JupyterLab is a full shell, so never publish it. It assumes Traefik v3, an entrypoint named https and a certificate for your host name. Create the network once if it is missing: docker network create proxy. In docker-compose.yml, delete the ports: block, then add:

services:
  unsloth:
    networks:
      - proxy
    labels:
      - "traefik.enable=true"
      - "traefik.docker.network=proxy"
      - "traefik.http.routers.unsloth.rule=Host(`unsloth.<your-domain>`)"
      - "traefik.http.routers.unsloth.entrypoints=https"
      - "traefik.http.routers.unsloth.tls=true"
      - "traefik.http.services.unsloth.loadbalancer.server.port=8000"
networks:
  proxy:
    external: true

Merge this into your existing file. Add traefik.http.routers.unsloth.tls.certresolver=<your-resolver> if you use ACME. Put an access layer in front (for example your SSO or Traefik basic auth), because Studio's tools can run code in the container.

Check it worked: curl -I https://unsloth.<your-domain>/ answers from Studio over HTTPS, and curl -m 5 http://<server-ip>:8888 is refused from another machine.

Without Traefik: Caddy#

Choose Caddy if you have no Traefik yet and run one small server. It is one small file and it gets the certificate for you. Only Studio is published; JupyterLab (port 8888) stays on 127.0.0.1.

The Caddyfile for Unsloth Studio (public domain, app on this machine at port 8000):

unsloth.example.com {
    basic_auth {
        myuser <paste-the-output-of-caddy-hash-password>
    }
    reverse_proxy localhost:8000
}

Home only (use a name your own DNS or hosts file points at this machine; .home.arpa is reserved for home networks):

unsloth.home.arpa {
    tls internal
    basic_auth {
        myuser <paste-the-output-of-caddy-hash-password>
    }
    reverse_proxy localhost:8000
}

Studio can run code in the container, so the Caddyfile puts a login in front. Make the hash with caddy hash-password and replace myuser with your own user name. Keep the 127.0.0.1:8000:8000 port from this guide.

Trust the root certificate (home only). On the Caddy machine run caddy trust. On every other device, copy the file pki/authorities/local/root.crt from Caddy's data folder and import it:

• Linux (Debian or Ubuntu): sudo cp root.crt /usr/local/share/ca-certificates/caddy-root.crt && sudo update-ca-certificates
• Windows (PowerShell as administrator): certutil -addstore -f ROOT root.crt
• Mac: sudo security add-trusted-cert -d -r trustRoot -k /Library/Keychains/System.keychain root.crt

Caddy's data folder is /var/lib/caddy/.local/share/caddy for the Linux service, %AppData%\Caddy on Windows and ~/Library/Application Support/Caddy on Mac (in the Docker setup below it is ./data/caddy).

Run Caddy. Pick one:

• Linux (official apt repository, Debian or Ubuntu): sudo apt install -y debian-keyring debian-archive-keyring apt-transport-https curl, then add the Caddy repository as shown on the Caddy install page, sudo apt update && sudo apt install caddy, put your site in /etc/caddy/Caddyfile and run sudo systemctl reload caddy.
• Windows (PowerShell): winget install CaddyServer.Caddy, save the Caddyfile in a folder, then caddy run --config Caddyfile from that folder.
• Mac: brew install caddy, save the Caddyfile, then caddy run --config Caddyfile.
• Docker: a separate folder ~/docker_volumes/caddy with this compose.yaml and the Caddyfile next to it. Remove the ports: block of Unsloth and add networks: [proxy] to it as in the Traefik block above (without the labels). Do not add port 8888 to Caddy.

services:
  caddy:
    image: caddy:2.11.7
    restart: unless-stopped
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - ./data:/data
    networks:
      - proxy
networks:
  proxy:
    external: true

For the Docker setup the Caddyfile points at the container name instead of localhost:

unsloth.example.com {
    basic_auth {
        myuser <paste-the-output-of-caddy-hash-password>
    }
    reverse_proxy unsloth:8000
}

Caddy passes WebSockets through by default, so nothing extra is needed for them.

Check it worked: curl -I https://unsloth.example.com/ answers HTTP/2 401 (the login prompt of basic_auth; with the right user and password you get Studio), and curl -m 5 http://<server-ip>:8888 is still refused from another machine. Caddy is pinned to 2.11.7, the current release when this guide was checked.

Security and testing#

Keep both ports on localhost, keep the passwords in .env, and treat Studio like a shell.

The docs list these risks, so plan for them:

• The container runs as root. Files it writes into data/work can be owned by root. Fix with sudo chown -R "$USER" ~/docker_volumes/unsloth/data/work
• Published ports listen on every interface unless you bind them. That is why this guide uses 127.0.0.1:
• JupyterLab is a full shell. Anyone who signs in can run any command in the container
• Studio's server-side tools are on by default and can run commands inside the container. Mount only folders you are happy to give it
• Values in .env show up in docker inspect for anyone with Docker access. Do not reuse these passwords elsewhere
• Do not use UNSLOTH_STUDIO_SECURE or the Cloudflare options to publish it. That makes a public link

Status checks:

docker compose ps
docker compose logs --tail 50 unsloth
docker exec unsloth nvidia-smi

Linux file permissions:

chmod 600 ~/docker_volumes/unsloth/.env
ls -l ~/docker_volumes/unsloth/.env

Troubleshooting#

Known bugs#

llama-server failed to start or "Check that the GGUF file is valid" (Windows Insider builds). After updating the native Windows app to v0.1.910-beta, every model failed to load on Windows Insider builds 26220, 26300 and 26340. This is GitHub issue 15022. The fix (pull request 15034) was merged on 25 October. Update the app. It does not affect the Linux container in this guide.

Diagnostic commands#

docker compose ps
docker logs unsloth --tail 100
docker inspect unsloth --format '{{.State.Status}}'

Updates and backups#

Update by changing the pinned tag on purpose, and keep a way back.

1. Read the release notes.
2. Write down the tag that works now (v0.1.910-beta).
3. Change image: in docker-compose.yml to the new tag. The image is about 10 GB, so pull on a good connection.
4. Pull and recreate:

cd ~/docker_volumes/unsloth
docker compose pull unsloth
docker compose up -d unsloth

Your models, files and Studio data carry over, because they sit in the folders and the named volume. To roll back, put the old tag back and run docker compose up -d again.

What to back up: docker-compose.yml, .env, data/work (your datasets and results) and the unsloth-studio volume. The Hugging Face cache can be downloaded again, so it is optional.

The Studio volume is a Docker named volume, so a small helper container copies it out. The archive includes .env, so keep it private.

Linux#

cat > ~/docker_volumes/unsloth/backup.sh << 'EOF'
#!/usr/bin/env bash
set -euo pipefail
cd ~/docker_volumes/unsloth
umask 077
tar czf "backups/unsloth-files-$(date +%F).tar.gz" docker-compose.yml .env data/work
docker run --rm -v unsloth-studio:/src:ro -v "$PWD/backups":/out alpine:3.20 \
  tar czf "/out/unsloth-studio-$(date +%F).tar.gz" -C /src .
find backups -name 'unsloth-*.tar.gz' -mtime +30 -delete
EOF
chmod +x ~/docker_volumes/unsloth/backup.sh
( crontab -l 2>/dev/null; echo "0 3 * * * $HOME/docker_volumes/unsloth/backup.sh" ) | crontab -

Windows (PowerShell)#

$d = "$HOME\docker_volumes\unsloth"
$stamp = Get-Date -Format yyyy-MM-dd
Compress-Archive -Path "$d\docker-compose.yml","$d\.env","$d\data\work" -DestinationPath "$d\backups\unsloth-files-$stamp.zip" -Force
docker run --rm -v unsloth-studio:/src:ro -v "${d}\backups:/out" alpine:3.20 tar czf "/out/unsloth-studio-$stamp.tar.gz" -C /src .

To schedule it daily at 03:00, save those lines as backup.ps1 and register it:

$a = New-ScheduledTaskAction -Execute "pwsh.exe" -Argument "-NoProfile -File $HOME\docker_volumes\unsloth\backup.ps1"
Register-ScheduledTask -TaskName "unsloth-backup" -Action $a -Trigger (New-ScheduledTaskTrigger -Daily -At 3am)

Use powershell.exe instead of pwsh.exe if you only have Windows PowerShell.

Mac (Terminal)#

The native app keeps its data in your user folder. Copy the Unsloth app data folder with Time Machine or your usual backup tool, and keep your datasets in a folder you already back up.

Restore test. Unpack the Studio archive into a throwaway volume and check it has your files:

docker volume create unsloth-restore-test
docker run --rm -v unsloth-restore-test:/dst -v ~/docker_volumes/unsloth/backups:/in alpine:3.20 \
  sh -c "tar xzf /in/unsloth-studio-$(date +%F).tar.gz -C /dst && ls /dst"
docker volume rm unsloth-restore-test

On Windows PowerShell, use the same docker lines with the paths from the backup step. If you see Studio's folders listed, the backup is good.

Performance and resources#

Speed and memory depend on your GPU, model size, context length and batch size. We did not measure them.

What the docs say:

• Unsloth claims 2x faster training with up to 70% less VRAM than standard methods, with no accuracy loss. That is the project's own claim.
• Minimum VRAM is in the table in Before you start. It is a floor, not a promise.
• Studio and JupyterLab share one GPU. A model loaded in Studio keeps its memory until you unload it. Unload the chat model before you train.
• unsloth gradient checkpointing (the default) cuts VRAM use a lot.
• The triton cache folder can be mounted to speed up restarts. This is optional.

Watch it live:

docker stats unsloth
nvidia-smi -l 2

FAQ#

Can I do this without an NVIDIA GPU? On Linux with AMD, use the separate unsloth/unsloth-rocm image (the docs say RDNA1 and newer, plus CDNA). On a Mac, use the native app. The docs say Unsloth also runs chat and Data Recipes on CPU, but training needs a GPU or Mac hardware.

How big a model can I train? Stay at 1-4B for a first run. By the docs' floor numbers, a 3B model needs at least 3.5 GB of VRAM with QLoRA, and a 14B model needs 8.5 GB. Real runs need headroom.

Do I lose my work when I remove the container? No, if you used the folders and the named volume from this guide. Anything else written inside the container is lost, so save results under /workspace/host, which is data/work on your disk.

Should I use latest instead of a pinned tag? Pinned is safer. The latest tag moves with every release, and all releases are betas. Update on purpose.

Is my data sent anywhere? Training runs in your container. Models download from Hugging Face the first time. Studio's web search and Cloudflare sharing are separate features. Do not turn on sharing if the data is private.

Sources#

All pages read on 2026-10-26:

• Unsloth Docker guide (updated September 2026)
• Unsloth Studio install
• Unsloth Studio quickstart (train, data, export)
• Export models with Studio
• Unsloth requirements and VRAM table
• Fine-tuning guide
• Saving to GGUF
• unslothai/unsloth on GitHub: README and releases
• Docker Hub: unsloth/unsloth
• GitHub issue 15022 (Windows Insider regression)
• Caddy docs: automatic HTTPS, tls internal, reverse_proxy, install

Checked against the official docs on 2026-10-26. Not run by DIY Lab Hub.