A 70-billion-parameter model needs roughly 140 GB of GPU memory in full precision. A 4 GB graphics card holds about 3% of that. The usual answer is quantization — shrink the weights until they fit — but that trades accuracy for size, and a 671-billion-parameter model stays out of reach no matter how aggressively you quantize.

AirLLM takes a different route. Instead of loading the whole model, it streams it one layer at a time: load a layer from disk, run it, unload it, load the next. The GPU only ever holds a single layer, so the memory a model needs is set by its largest layer, not by its total size. That is how the project runs a 70B model on a 4 GB card, a 405B model on 8 GB, and a 671B model on 12 GB — in full precision, with no quantization and no pruning. The project, lyogavin/airllm, has 42,466 stars and 4,391 forks, gained 6,055 stars in the last month, and sits at #23 on GitHub's monthly trending list for AI. Version 4.0.0 landed on PyPI on September 5, 2026, adding support for the 2.8-trillion-parameter Kimi K3.

This tutorial runs it end to end. You will install AirLLM 4.0.0, load a small model, watch it split the weights into per-layer shards, ask it three questions, and measure exactly how much memory it uses versus plain Transformers — all on a CPU-only machine with no GPU at all. Every command and every quoted output below ran in our sandbox on November 15, 2026.

What you'll need#

  • Python 3.9 or later (3.12 used here).
  • pip, and about 2 GB of free disk for the per-layer shards — the first run splits the model into shards roughly the size of the model itself, so you need room for the model twice over, briefly.
  • No account, no API key, no GPU. The whole tutorial runs on CPU. A GPU makes it faster; the code is identical, except you pass device="cuda:0" (the default) instead of device="cpu".
  • A Hugging Face model ID. We use Qwen/Qwen2.5-0.5B-Instruct (954 MB) because it is small enough to verify on CPU. The same steps work for Qwen3-32B, Llama 3.x 70B, and the rest — the only difference is download size and speed.

1. Install AirLLM#

AirLLM is a plain PyPI package. Pin the version — 4.0.0 is what every output below comes from — and let pip pull its dependencies:

pip install airllm==4.0.0

That installs airllm plus torch>=2.4, transformers>=4.49,<6, accelerate>=1.0, safetensors, huggingface-hub, scipy, and sentencepiece. On a CPU-only machine, use the CPU torch wheel to avoid downloading the ~800 MB CUDA build:

pip install torch --index-url https://download.pytorch.org/whl/cpu
pip install airllm==4.0.0

Verify the install:

python3 -c "import airllm; print('airllm OK')"

Our sandbox answered airllm OK with torch 2.15.0+cpu and transformers 5.21.0. One gotcha worth knowing before you start: AirLLM's AutoModel.from_pretrained defaults to device="cuda:0". On a CPU-only install, pass device="cpu" explicitly — otherwise the first tensor move fails with AssertionError: Torch not compiled with CUDA enabled. That is the only CPU-specific change in this tutorial.

2. Run your first model — and watch it split the weights#

AirLLM's API mirrors Transformers. The difference is what happens on first load: it splits the model into one file per layer, under the directory you name, and reuses those shards on later runs. Run this:

from airllm import AutoModel

model = AutoModel.from_pretrained(
    "Qwen/Qwen2.5-0.5B-Instruct",
    layer_shards_saving_path="./airllm-shards",
    device="cpu",
)

Our run answered:

using generic AirLLM streaming model for architecture: Qwen2ForCausalLM
saved layers already found in /var/tmp/airllm-shards/splitted_model
built empty Qwen2ForCausalLM via AutoModelForCausalLM (attn=sdpa)
LOAD 1.8s device=cpu peakRSS=479MB

The load took 1.8 seconds and the process held 479 MB of RAM — the model itself is 954 MB, and it is not fully in memory. The split wrote 52 files totalling 988.1 MB into airllm-shards/splitted_model/: one .safetensors per layer plus a .done marker each. The embedding table is the big one at 272 MB; each of the 24 transformer layers is about 29.8 MB. That is the whole trick made visible: the largest single file the GPU (or CPU) ever has to hold is the 272 MB embedding, not the 954 MB model.

On the first run the split itself takes a minute or two while it reads the model and writes the shards. On later runs AirLLM prints saved layers already found and skips straight to loading. The shards are a cache, not a copy you need to manage — but they do cost disk, roughly the size of the model, on top of the original download. The project's FAQ warns that the split is disk-consuming, and that a MetadataIncompleteBuffer error means you are out of disk.

Diagram of a hard drive feeding glowing model layer slabs one at a time into a GPU chip
One layer at a time: the disk holds the model, the GPU holds a single layer. Illustration generated for AI Frontier Post.

3. Ask it questions#

Load the model once, then generate with the chat template. The generate call is the same shape as the README's quickstart — use_cache=True and return_dict_in_generate=True — with max_new_tokens capped so a CPU run finishes in a reasonable time:

tok = model.tokenizer
messages = [{"role": "user", "content": "What is the capital of France?"}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
inputs = tok(prompt, return_tensors="pt", return_attention_mask=False,
             truncation=True, max_length=128, padding=False)
out = model.generate(
    inputs["input_ids"].to(model.device),
    max_new_tokens=40,
    use_cache=True,
    return_dict_in_generate=True,
)
new_ids = out.sequences[0][inputs["input_ids"].shape[1]:]
print(tok.decode(new_ids, skip_special_tokens=True).strip())

Three questions, verbatim from our run:

Q: What is the capital of France?
A: The capital of France is Paris.
  (8 tokens in 62.7s = 0.13 tok/s, peakRSS=785MB)

Q: Explain in one sentence why the sky is blue.
A: The sky appears blue because of the scattering of sunlight by tiny
water droplets or ice crystals in the atmosphere. This process causes
light to be scattered and bent as it passes through the air, resulting in
  (40 tokens in 325.0s = 0.12 tok/s, peakRSS=788MB)

Q: Write a Python function that returns the square of a number.
A: Certainly! Below is a Python function that takes an integer as input
and returns its square:

```python
def square_number(n):
    """
    Returns the square of the given number.

    Parameters:
  (40 tokens in 297.0s = 0.13 tok/s, peakRSS=788MB)

Two things to notice. First, the answers are correct as far as they go — but the second and third hit the 40-token cap and stop mid-sentence and mid-docstring. That cap is a choice for this CPU demo, not a limit of the tool; raise max_new_tokens and the model finishes the thought. Second, the 0.5B demo model's science is shaky: the sky is blue because of Rayleigh scattering off air molecules, not "water droplets or ice crystals." Treat the small model as a way to verify the mechanism, not as an oracle. Use a larger model for answers you will rely on.

And the speed: about 0.12–0.13 tokens per second on CPU. A 40-token answer takes five minutes. That is the honest CPU number, and it is slow — the next section explains why, and why it is not the number that matters.

4. Measure the memory win#

The same model, the same three questions, loaded the ordinary way with plain Transformers — whole model in RAM, no streaming. The comparison, all on the same CPU-only machine:

AirLLMPlain Transformers
Load time1.8 s9.3 s
Peak RAM after load479 MB487 MB
Peak RAM during generation788 MB1,405 MB
"Capital of France" (8 tokens)62.7 s · 0.13 tok/s4.9 s · 1.64 tok/s
"Why is the sky blue" (40 tokens)325.0 s · 0.12 tok/s15.2 s · 2.63 tok/s
"Python square function" (40 tokens)297.0 s · 0.13 tok/s15.7 s · 2.54 tok/s

AirLLM peaks at 788 MB where Transformers peaks at 1,405 MB — 44% less RAM for the same 0.5B model. But it is about twenty times slower on CPU. That is the trade, stated plainly: layer streaming buys memory with time. Every generated token re-streams all 24 layers from disk, and on a CPU the per-layer overhead dominates. On a GPU the balance shifts — the layer loads overlap with fast GPU compute, which is why the tool is designed for small GPUs, not for CPUs. The CPU run proves the mechanism works; it is not the configuration you would choose for real use.

Why does the memory win look modest here? Because the model is tiny. The 0.5B weights are only 954 MB, so most of both processes' memory is Python, torch, and the KV cache, not the weights. The win scales with the model: for a 70B model, Transformers needs the whole ~140 GB in memory, while AirLLM needs roughly one layer at a time — about 4 GB. The 0.5B demo shows the shape of the saving; the big models are where it becomes the difference between running and not running.

5. Keep a reusable ask script#

Wrap the load-and-generate loop in a small CLI so you can ask questions without rewriting the boilerplate. Save this as ask_airllm.py:

#!/usr/bin/env python3
"""Ask a local model a question using AirLLM's layer-streaming inference."""
import argparse, os, sys, time

os.environ.setdefault("HF_HUB_DISABLE_XET", "1")

def main():
    ap = argparse.ArgumentParser(description="Ask a local model via AirLLM.")
    ap.add_argument("question", help="Question to ask the model.")
    ap.add_argument("--model", default="Qwen/Qwen2.5-0.5B-Instruct",
                    help="Hugging Face model ID (default: %(default)s).")
    ap.add_argument("--max-new-tokens", type=int, default=120)
    ap.add_argument("--shards-dir", default="./airllm-shards",
                    help="Where AirLLM stores the per-layer shards.")
    args = ap.parse_args()

    from airllm import AutoModel

    t0 = time.time()
    model = AutoModel.from_pretrained(args.model,
        layer_shards_saving_path=args.shards_dir, device="cpu")
    load_s = time.time() - t0

    tok = model.tokenizer
    messages = [{"role": "user", "content": args.question}]
    prompt = tok.apply_chat_template(messages, tokenize=False,
                                     add_generation_prompt=True)
    inputs = tok(prompt, return_tensors="pt", return_attention_mask=False,
                 truncation=True, max_length=512, padding=False)
    t1 = time.time()
    out = model.generate(inputs["input_ids"].to(model.device),
                         max_new_tokens=args.max_new_tokens,
                         use_cache=True, return_dict_in_generate=True)
    gen_s = time.time() - t1
    n_new = out.sequences.shape[1] - inputs["input_ids"].shape[1]
    new_ids = out.sequences[0][inputs["input_ids"].shape[1]:]
    print(tok.decode(new_ids, skip_special_tokens=True).strip())
    print(f"[loaded {args.model} in {load_s:.1f}s; "
          f"{n_new} tokens in {gen_s:.1f}s ({n_new / gen_s:.2f} tok/s) "
          f"on {model.device}]", file=sys.stderr)

if __name__ == "__main__":
    main()

Run it:

python3 ask_airllm.py --max-new-tokens 30 "What is the capital of France?"

Our run answered:

The capital of France is Paris.
[loaded Qwen/Qwen2.5-0.5B-Instruct in 2.1s; 8 tokens in 90.7s (0.09 tok/s) on cpu]

The answer goes to stdout, the timings to stderr, so you can pipe the answer somewhere else. Swap --model for any Hugging Face ID AirLLM supports — Qwen3-32B, Llama 3.x 70B, DeepSeek-V3 — and the first run splits that model into shards under --shards-dir. Later runs reuse them. One decoding note that cost us a debugging round: decode only the newly generated tokens (out.sequences[0][inputs["input_ids"].shape[1]:]), not the full sequence sliced by the prompt's character length. The tokenizer's decoded prompt and the prompt string differ in length, so slicing by len(prompt) silently drops the answer and leaves an empty string.

Going bigger: what changes on a real GPU#

Everything above ran on a CPU, because that is what this sandbox has. The numbers that make AirLLM worth using are the GPU ones — and those come from the project's own measurements, not from this run, so treat them as the project's claims on named hardware, not as verified here. From the project's README, measured VRAM on a single GPU:

ModelVRAM on one GPU
Qwen3 / Mistral / Phi, ~8B~1–2 GB
Qwen3-30B / Mixtral~1–3 GB
Qwen3.8-27B3.33 GB
Qwen3-235B~3 GB
Llama 3.x 70B~4 GB
Qwen3.8-Flash-Next, ~180B5.95 GB (RTX 4090)
Llama 3.1 405B~8 GB
DeepSeek-V3 671B~12 GB
Kimi K3 2.8T3.72 GB (RTX 6000 Ada)

The pattern is the point: VRAM tracks the largest layer, not the model. A 2.8-trillion-parameter model needs 3.72 GB because its layers are small; a 405B model needs 8 GB because its layers are bigger. That is why the headline number — 70B on a 4 GB card — is not a special case. It is the same mechanism at every scale.

Two newer capabilities from the project's 2026 updates are worth knowing. Training support landed in September 2026: Qwen3.8-Flash-Next 125B trains under 6 GB and Qwen3.8-27B trains at about 2 GB with a 512-token sequence, using the same layer-streaming idea in reverse. And 4-bit/8-bit compression via bitsandbytes is available — the README claims up to a 3x speedup — but it needs the bitsandbytes package, and on a CPU-only install it fails outright. Our test:

COMPRESSION 4bit: ImportError: WARNING: bitsandbytes not found. Compression
needs bitsandbytes. To use compression, please install bitsandbytes:
`pip install bitsandbytes`

Install bitsandbytes if you want compression, and note that its 4-bit kernels need a CUDA GPU — compression is a GPU feature in practice, not something to reach for on CPU.

A server rack running a large brain-network model beside a laptop running stacked model layers
A 671B model on a server, a 70B model on a laptop — the same code, different hardware. Illustration generated for AI Frontier Post.

When to use this vs alternatives#

Use AirLLM when the model is bigger than your GPU and you want full precision. That is the case it is built for: a 70B model on a 4 GB card, a 405B model on 8 GB, with no quantization and no pruning. The cost is speed — on CPU it runs at about 0.12 tokens per second, and even on a GPU the per-token layer streaming is slower than a model that sits fully loaded in VRAM. Use it for interactive exploration, for running a model you could not otherwise load at all, or for a single-GPU workstation where the alternative is not running the model.

Use llama.cpp or Ollama when you want fast local inference and accept quantization. GGUF-quantized models run well on CPU and on small GPUs, and Ollama makes them a one-command install. You trade a little quality for the quantization, and you need a quantized build of the model — but for everyday chatting and coding help, that trade is usually the right one, and the speed is an order of magnitude better than AirLLM on CPU.

Use Transformers with device_map="auto" when the model fits in RAM plus VRAM combined. Accelerate loads the whole model and offloads the layers that do not fit in VRAM to CPU RAM. It is simpler — no re-sharding step, no extra disk — and it keeps the fast path for the layers that do fit on the GPU. It only fails when the model is bigger than your RAM and VRAM together, which is exactly where AirLLM's disk streaming takes over.

Use vLLM when you are serving a model to many users. vLLM is built for throughput: continuous batching, paged attention, high requests per second. It needs the model (or most of it) in GPU memory, so it is not the tool for a 4 GB card and a 70B model — but if you have the VRAM, it will serve that model far faster than AirLLM's one-layer-at-a-time streaming.

Use Strata when you want another way to run a very large model on consumer hardware. We covered Strata, which runs a 125B model on a gaming PC, in October. It takes a different approach to the same problem — a large model on hardware that should not be able to hold it — and the two are worth comparing on the same machine before you commit to one.

Limitations and honest caveats#

  • CPU is a demo, not a configuration. At 0.12–0.13 tokens per second, a 40-token answer takes five minutes. If you have no GPU, use llama.cpp or Ollama instead. AirLLM's value is on a small GPU.
  • The first load doubles your disk usage, briefly. The split writes a full copy of the model as per-layer shards (988 MB for a 954 MB model). You need the model on disk twice over. Delete the shards and the original and you have to re-split.
  • Per-token latency is higher than a loaded model. Every token re-streams every layer. That is the price of the memory saving, and it does not go away on GPU — it just matters less, because the compute is fast.
  • The big-model GPU numbers are the project's, not ours. The 70B-on-4GB, 405B-on-8GB, and 671B-on-12GB figures come from the AirLLM README's own measurement table. We verified the mechanism (install, split, generation, memory) on CPU with a 0.5B model; we did not run a 70B model on a 4 GB card here, because this sandbox has no GPU.
  • The small demo model's answers are unreliable. Our 0.5B model said the sky is blue because of "water droplets or ice crystals." It is Rayleigh scattering off air molecules. Use a larger model for answers you will rely on.
  • Compression needs bitsandbytes and a CUDA GPU. The 4-bit/8-bit path fails on a CPU-only install without bitsandbytes, and its kernels are a GPU feature.
  • Training support is new and unverified here. The September 2026 training mode (125B under 6 GB) is a project claim; we did not test it.

The takeaway#

The VRAM a model needs is not set by its size. It is set by its largest layer — if you are willing to stream the rest from disk, one layer at a time. AirLLM is the cleanest implementation of that idea we have tested: a PyPI install, a three-line load, a per-layer split you can watch happen, and a memory footprint that stays near one layer while the model itself can be hundreds of billions of parameters. The cost is speed, and on CPU that cost is severe — 0.12 tokens per second is a demo, not a workflow. But on the hardware the tool is actually for — a 4 GB card that should not be able to hold a 70B model — the trade is the whole point. Install it, split a model, and the card you thought was too small stops being the limit.

Sources: the AirLLM repository and airllm 4.0.0 on PyPI. Star counts, fork counts, and trending placement as observed November 15, 2026; version 4.0.0 and all command outputs verified by running them in our sandbox. GPU VRAM figures are the project's own measurements from its README, not verified here.