Run a 744-billion-parameter model on your own machine: a hands-on guide to Colibri

A hummingbird perched on a branch — colibri means hummingbird: tiny engine, immense model — The namesake: a hummingbird — tiny engine, immense model. Photo: Pexels (free stock photo).

Frontier open models have outgrown ordinary hardware. GLM-5.2 has 744 billion parameters — hundreds of gigabytes on disk — so running it sounds like a cluster problem. Colibri (JustVugg/colibri, past 41,000 GitHub stars, Apache-2.0) observes that a mixture-of-experts model uses only a small slice of itself for each token: about 5.4% of parameters are active per token, and for GLM-5.2 only around 11 GB of routed experts change from one token to the next. So colibri keeps the dense part (attention, shared experts, embeddings) in RAM and streams the routed experts from disk through a cache that learns which experts your work uses. A pure-C engine, zero dependencies, no GPU required. This tutorial installs it, surveys the model catalog, chats with a model, and serves it behind an OpenAI-compatible API.

What you’ll need

A computer — 8 GB of RAM at the very least (16 GB or more is better), 22 GB of free disk for the smallest model, and an internet connection. Linux, macOS, or Windows. A graphics card is optional.
A fast disk if you want the big models. For the streaming models the disk sets the speed — an NVMe drive helps most.
Bandwidth for the download. 23 GB for the smallest chat model; 429 GB for GLM-5.2; 1.56 TB for Kimi K3. Downloads resume where they stopped.
git and python3 (plus build-essential on Linux). The setup prints the exact install command if anything is missing.

Step 1 — run the one-step setup

# Debian / Ubuntu first:
sudo apt install git python3 build-essential

git clone https://github.com/JustVugg/colibri && cd colibri
./start-here.sh

On macOS: xcode-select --install and brew install libomp git python, then the same two lines. On Windows, download the ZIP and double-click START-HERE.bat (it offers to install Python if it's missing). The setup does five things: it surveys your machine (RAM, free disk, CPU, GPUs), recommends the most capable model that fits — the always-resident part plus a minimum cache of experts must fit in your RAM, the download on your disk —, builds the engine for your machine (or downloads a prebuilt one: CPU always, Vulkan or CUDA for a GPU), downloads the model with progress and resume, then starts colibri and opens the dashboard. Run it again later and it just starts — no rebuild, no re-download. If you use an AI coding assistant, you can also have it do all of this: the repo ships docs/AI_SETUP.md with every step as a command, plus an MCP server (coli mcp) that can detect hardware, recommend, install, start, and stop.

Step 2 — survey the model catalog

./start-here.sh --list

This shows every supported model against your machine. RAM is two numbers: below the first the model does not start, from the second it runs as measured. Some highlights from the project's own benchmark table:

Qwen3.6-35B-A3B (--model qwen36-35b): 23 GB download, 10/20 GB RAM, ~6 tok/s on the CPU — the sane first model.
DeepSeek V4 Flash, 284B (--model deepseek-v4-flash): 167 GB, 16/32 GB, 0.93 tok/s on a Ryzen 7 5800X.
GLM-5.2, 744B (--model glm-5.2): 429 GB, 16/24 GB, 0.05–0.1 tok/s cold on a 25 GB laptop — a demo at that speed, not a chatbot.
Kimi K3, 2.8T (--model kimi-k3): 1.56 TB, 32/64 GB, about 9.4 s per token.

Install one with ./start-here.sh --model qwen36-35b, or pass --yes to take the recommendation with no questions. Other useful options: --dir DIR to put the models on another disk, --backend cpu to skip the GPU entirely, --reconfigure to switch models later.

Step 3 — chat in the terminal

./coli chat

The banner tells you the whole story, e.g. colibri v2.0.0 · GLM-5.2 · 744B MoE · int4 · streaming CPU / √ ready in 32s · resident 9.9 GB. That last number is the point: the entire 744B model lives on disk, under 10 GB sits in RAM. c/coli status shows what's installed and whether it runs; c/coli stop stops it. Handy extras on the same launcher: ./coli plan --model /nvme/qwen36 previews where the model will live (VRAM, RAM, disk), ./coli doctor is a read-only readiness check, and ./coli tune measures and saves this machine's fastest safe settings.

A hummingbird feeding at a flower, wings a blur — only the experts the router asks for leave the disk — Only the routed experts leave the disk — the cache learns which ones your conversations use. Photo: Pexels (free stock photo).

Step 4 — serve it to your other apps

./coli serve            # API + dashboard, no browser

The setup already starts this server for you; it prints the addresses other apps can use. It is one server with several APIs: OpenAI-compatible (/v1/chat/completions, /v1/completions, /v1/models — with streaming, JSON replies, stop sequences, and logprobs), Anthropic-compatible (/v1/messages, so Claude Code and the Anthropic SDKs work against it), tool calling on every chat engine except Inkling and OLMoE, and image input on the vision-capable models. Try it:

curl http://127.0.0.1:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "<the id `coli status` prints>",
       "messages": [{"role": "user", "content": "In one sentence: why can a mixture-of-experts model stream its experts from disk?"}]}'

Coding CLIs and editors connect as to any OpenAI-compatible provider: base URL http://127.0.0.1:8000/v1, the model id coli status prints, any non-empty key.

Step 5 — watch the router work in the dashboard

./coli web              # API + dashboard, opens a browser

The dashboard also lives at http://127.0.0.1:8000/. Beyond chat, it has the System One page (give the model a document and the only answers it may pick — you get a probability per allowed answer), the Profiling page (where each turn spends its time, phase by phase), and the Brain: the measured expert atlas of GLM-5.2, 13,260 characterized experts in ten regions, with live routing that flashes every expert your turn actually fired. It's the clearest visual explanation of why the model is small per token.

A hummingbird hovering in mid-air — on the Brain page, one cell per expert flashes as the router fires — The Brain page: one cell per expert, colored by storage tier, flashing as the router fires. Photo: Pexels (free stock photo).

What you built

A local inference stack for frontier mixture-of-experts models: one-command install, terminal chat, an OpenAI- and Anthropic-compatible API on 127.0.0.1:8000, and a dashboard that shows the MoE router working in real time — all on hardware you already own.

Honest limitations

The 744B headline is a demo at laptop scale. 0.05–0.1 tokens per second on a 25 GB laptop means 10–20 seconds per token. The everyday-usable models are the smaller ones: Qwen3.6-35B at ~6 tok/s on CPU (~9.9 with Vulkan on integrated graphics).
The disk is the speed limit. For streaming models, how fast experts arrive from the disk sets the pace — an NVMe helps most, an ordinary disk is slower, and RAM still matters for the resident dense part plus the expert cache.
The downloads are enormous. 429 GB for GLM-5.2, 1.56 TB for Kimi K3 — resumable, but plan the bandwidth and the free space.
Only MoE architectures benefit. Dense models have no experts to stream; the trick works because each token activates ~5.4% of the model. The engine covers thirteen families — ten chat, one image model, and decision models.
GPU builds can fall back. If a build fails (e.g. the installed CUDA toolkit no longer supports the card), the setup picks the next backend itself — Vulkan, then CPU — and ./start-here.sh --backend cpu forces CPU from the start.