Install llama-swap on Ubuntu (the Easy Way)

If you run local models with llama.cpp, you already know the annoyance: llama-server serves one model at a time. Want a different model? Kill it, restart with new flags. llama-swap fixes that. It’s a tiny Go proxy that sits in front of your models, reads the model field from each request, and starts (or swaps to) the right one automatically. One OpenAI-compatible endpoint, every model behind it, zero Python.

It’s a single static binary, so getting it running takes about five minutes. Here’s how I set it up on my homelab server.

Install

llama-swap is a single static binary, so installing it is really just “download it and put it on your PATH.” Here’s the normal way first, then the shortcut I personally use.

The normal way (recommended)

Head to the releases page and grab the Linux build for your architecture — amd64 for most desktops and servers, arm64 for a Pi. Extract it and drop the binary somewhere on your PATH:

VERSION=v232   # whatever the current release is
curl -fL -o llama-swap.tar.gz \
  "https://github.com/mostlygeek/llama-swap/releases/download/${VERSION}/llama-swap_${VERSION#v}_linux_amd64.tar.gz"
tar -xzf llama-swap.tar.gz
mkdir -p ~/.local/bin && mv llama-swap ~/.local/bin/
llama-swap --version

To update later, you do the same thing with a newer version number. There are also official Docker images if you’d rather run it in a container — the repo’s README covers those.

If llama-swap isn’t found afterward, add ~/.local/bin to your PATH: put export PATH="$HOME/.local/bin:$PATH" in your ~/.bashrc and reload.

The shortcut I use (optional)

Doing that dance every few days got old — releases move fast — so for myself I wrote a small script that automates exactly the steps above. It figures out your OS and architecture, pulls the matching binary straight from the official releases page, drops it in ~/.local/bin, and restarts the systemd service if you have one. Install and update end up being the same single command.

There’s nothing clever in it — it’s just a wrapper around the official download, and you can read the whole thing here before running anything. If it saves you time too, great; if you’d rather do it by hand, the section above is all you need.

curl -fL -o llama-swap-install.sh \
  https://gist.githubusercontent.com/aguyintech/902120a3083fceca0ee2e90cddf7fa50/raw/llama-swap-install-or-update.sh

chmod +x llama-swap-install.sh

./llama-swap-install.sh latest    # or pin a version: ./llama-swap-install.sh v232

Updating later is the same command with a newer version — that’s the whole appeal.

A minimal config

llama-swap is driven by one YAML file. Each model gets a name and the command that launches it. Two things make life easier: the ${PORT} macro (assigned automatically, so you never fight over ports), and a macros: block to hold the flags every model shares — so each model block stays short.

~/llama-swap/config.yaml:

# Big models can be slow to load on first request — give the health check room.
healthCheckTimeout: 300

# Shared base for every model. Reference it with ${server} below.
macros:
  "server": >
    /home/YOU/llama.cpp/build/bin/llama-server
    --n-gpu-layers 99
    --flash-attn on
    --host 0.0.0.0
    --port ${PORT}
    --jinja

models:
  "gemma-4-12b":
    cmd: |
      ${server}
      --model /home/YOU/models/gemma-4-12b.gguf
      --ctx-size 65536
      --cache-type-k q4_0
      --cache-type-v q4_0

  "qwen3-35b":
    cmd: |
      ${server}
      --model /home/YOU/models/qwen3-35b.gguf
      --ctx-size 98304
      --cache-type-k q8_0
      --cache-type-v q8_0

Everything shared lives in ${server}; everything model-specific — the model path, context size, KV-cache quantization, sampling — stays in the per-model block. Quantizing the KV cache (--cache-type-k/v to q8_0 or q4_0) is the easy win for fitting a bigger context into VRAM.

Use absolute paths. llama-swap launches your command directly, not through a shell, so ~/ is taken literally and won’t expand. Write /home/YOU/... everywhere — the binary and every model path. This trips up almost everyone once.

Run it as a service

So it survives reboots. Create /etc/systemd/system/llama-swap.service:

[Unit]
Description=llama-swap
After=network.target

[Service]
Type=simple
User=YOU
WorkingDirectory=/home/YOU/llama-swap
ExecStart=/home/YOU/.local/bin/llama-swap --config /home/YOU/llama-swap/config.yaml --listen 0.0.0.0:8092
Restart=on-failure
RestartSec=3

[Install]
WantedBy=multi-user.target

Replace YOU everywhere (a leftover YOUR_USERNAME gives you a cryptic status=200/CHDIR — ask me how I know). Then:

sudo systemctl daemon-reload
sudo systemctl enable --now llama-swap.service
sudo systemctl status llama-swap.service

active (running) means you’re live at http://YOUR-SERVER:8092/v1. Point any OpenAI-compatible tool at that URL, set model to one of your config keys, and llama-swap handles the rest.

Poke around the UI

Two web pages worth knowing:

  • The dashboard — just open http://YOUR-SERVER:8092/ in a browser. You get the list of models, which one is loaded, live logs, and load/unload controls. Handy for watching a model spin up or manually unloading one.
  • A specific model’s chat UI — llama.cpp ships its own chat interface, and llama-swap proxies to it per model at http://YOUR-SERVER:8092/upstream/<model-name>/. So .../upstream/gemma-4-12b/ drops you straight into that model’s built-in chat — perfect for a quick sanity test without wiring up a client.

Bonus: unload models when idle

By default a model stays in memory until you request a different one. On a single GPU that’s usually fine, but if you want VRAM freed after a quiet spell, add a ttl (seconds) to any model:

  "qwen3-35b":
    ttl: 300        # unload after 5 min idle
    cmd: |
      ${server}
      ...

The catch: the first request after an unload pays the reload cost (weights back into VRAM), then it’s fast again while warm. So I keep my daily driver with a long ttl or none at all, and put a short ttl on the big models I only reach for occasionally.

Bonus: vision models and the mmproj file

This one’s more llama.cpp than llama-swap, but it bites people here. A vision-capable model needs more than its GGUF — it also needs the multimodal projector, an mmproj-*.gguf file, passed with --mmproj.

A pattern I like: register the same base model twice under different names — one text-only, one with the projector — so I can reach for vision only when I actually need it (the encoder isn’t free):

  "qwen-35b":            # text only
    cmd: |
      ${server}
      --model /home/YOU/models/qwen-35b.gguf
      --ctx-size 98304
      --cache-type-k q8_0
      --cache-type-v q8_0

  "qwen-35b-vision":     # same model + projector
    cmd: |
      ${server}
      --model /home/YOU/models/qwen-35b.gguf
      --mmproj /home/YOU/models/mmproj-F16.gguf
      --ctx-size 98304
      --cache-type-k q8_0
      --cache-type-v q8_0
      --no-mmproj-offload

By default the projector offloads to the GPU, which is fastest. But --no-mmproj-offload keeps it on the CPU instead — and once you’re running a big context on a 35B model, that reclaimed VRAM is often exactly the headroom you need to fit the context at all. That’s the trade: a little image-processing speed for room to breathe.

That’s it. One binary, one config, one script to keep it fresh. Point your tools at the endpoint and forget about it — which is exactly what a proxy like this should let you do.

Related Posts