Inside-llm shows you exactly where two quantizations of the same model part ways — token by token
Picking a quantized model usually means trusting a single number — a perplexity score, a benchmark table, a forum post that says the 4-bit version is “fine.” A new open-source tool called inside-llm turns that choice into something you can watch: load two quantizations of the same model, and it shows you, token by token, exactly where the cheaper one first goes wrong.
- The tool: inside-llm, by Manuel García (u/AlternativeEnough352), posted to r/LocalLLaMA on November 15, 2026 at 4:46 PM EST — 337 upvotes and 34 comments at the time of writing.
- The demo: a 1:06 screen recording compares Qwen3.5-0.8B Q4_K_M against Q8_0 on the prompt “The capital of France is”, showing per-token probabilities and divergence as the text is generated.
- The split: in the longer demo on the project page, Qwen3.5-4B Q2_K picks “six” where Q8_0 picks “eight” at position 31 — divergence 0.395 bit, with entropy of 1.41 bits for the 2-bit model against 0.45 bits for the 8-bit one.
- What you see: per-token probabilities, the full candidate distribution with entropy, Jensen-Shannon divergence and KL in both directions, the sampler chain stage by stage, and a logit lens showing what each layer would predict.
- How to run it: self-contained Linux and Windows packages with CUDA or Vulkan;
inside-llm -a A.gguf -b B.ggufopens the model picker in your browser. Local only on 127.0.0.1 — no accounts, no telemetry. MIT license. - The limits: both models must share a tokenizer and fit in VRAM at the same time; text only, no split GGUFs, no embedding models — and no attention view, because llama.cpp has no API to extract it.


Ask anyone who runs models locally which quantization to pick and you will get a shrug dressed up as advice: the 4-bit is fine, the 8-bit is safer, just look at the perplexity. The problem is that a single number hides the only question that actually matters — where does the smaller model start to disagree, and does it matter there? A wrong token in a throwaway word is noise. A wrong token in a number, a name, or a fact is the whole answer going sideways.
That is the gap inside-llm is built to close. Manuel García, posting as u/AlternativeEnough352, put the tool on r/LocalLLaMA on Sunday afternoon, and the thread did what LocalLLaMA does best with a genuinely useful demo: it stayed, asked sharp questions, and started arguing about what the pictures actually mean.
What inside-llm actually shows you#
The core view is disarmingly simple. Load two GGUF models — two quantizations of the same model, a base and a fine-tune, or two sizes of one family that share a tokenizer — and the tool generates text in two columns, one per model, with brightness standing in for the probability of each token. Wherever the two models would pick differently, a magenta mark flags the split. Two modes: lockstep, where both models read the same text so the per-position comparison stays clean, and free, where each follows its own picks until they part ways.
Around that core sit four more views. Candidates shows the full distribution behind each pick — mirrored bars for model A and B, entropy, the effective number of options, Jensen-Shannon divergence and KL in both directions. Gates traces the sampler chain stage by stage — penalties, top-k, top-p, min-p, temperature, GBNF grammar — showing how many candidates survive each stage, with the chain order and seed configurable. Timeline plots entropy and divergence along the whole text, plus the tree of branches. And branching lets you click any candidate and continue generation from there; jumping back to a node is exact, because each model keeps KV-state snapshots and nothing is regenerated.
There is also a logit lens (--lens): what each layer of the model would predict, shown as a 2D map and as a 3D relief. The model picker takes models from a folder or searches Hugging Face and downloads two quants of a repo in one click, only offering B models with the same tokenizer as A and estimating whether the pair fits in VRAM.

The moment the two models split#
The 66-second video on the project page is the clearest version of the pitch. Asked how many legs a spider has, Qwen3.5-4B Q2_K and Q8_0 agree for thirty tokens — and then, at position 31, the 2-bit model picks “six” while the 8-bit model picks “eight.” The numbers behind the split are the point: the 2-bit model was only 57% sure of “six,” the 8-bit model 91% sure of “eight,” and the entropy gap (1.41 bits against 0.45) says the cheaper model was not just wrong, it was unsure. Divergence at that position measured 0.395 bit, with KL of 7.80 bits one way and 1.33 the other.
The Reddit video is the same idea at smaller scale: Qwen3.5-0.8B Q4_K_M against Q8_0, answering “The capital of France is,” with the per-token probabilities and the divergence visible as the sentence forms. Sessions can be recorded and replayed later as a static page — no engine, no model, no GPU needed to watch one back.

How it works under the hood#
Under the interface is one C++17 process linking llama.cpp directly, reading the full logits of both models on the same context at every step, with its own seeded sampler so runs are reproducible. Compact binary frames travel over WebSocket to a Svelte 5 and Canvas 2D page; three.js is only pulled in for the 3D relief, and only when you open it. Because it reads full logits rather than sampled text, the distributions, the sampler stages and the per-layer predictions are the real numbers, not reconstructions.
What it can’t do — and what the thread pushed back on#
García is upfront about the limits, and they are worth knowing before you install it. The two models must share a tokenizer. Both must fit in VRAM at the same time, only one GPU is used, integrated GPUs are never used and the tool falls back to CPU without a dedicated card. It is text only, it does not handle split (multi-part) GGUF files or embedding models, and it does not show attention — the project page is explicit that llama.cpp simply has no API to extract it.
The thread’s reception was warm but not uncritical. The top comment, from u/eze-paz (41 upvotes), put it plainly: “We need more of this kind of stuff to help see under the hood and stop believing AI is sentient magic.” Another called it “like a benchmark tool in real time,” and u/Evening-Blueberry singled out lockstep mode as the part they would actually use, noting that most quant comparisons stop at a perplexity number — then asked the sharp follow-up: does divergence bunch up on certain kinds of tokens, like numbers and names, or is it spread out?
The pushback is philosophical as much as technical. “Observe != understand,” wrote u/_underlines_ — and García conceded the point in part, replying that it “could be the first approach” and that visual things help a lot when it comes to learning. u/DrinkLimp9395 went further: “The true ‘honest limitation’ is that no one knows what to make of this data. Like, yeah, we can see that it diverged after this many tokens. Now what?” That one went unanswered, as did a commenter asking whether the tool is “just a complete clone of what Google already released” — a claim the thread never substantiated.
What to watch#
The open question is whether seeing the divergence changes which quantization people actually run. A perplexity table tells you the 2-bit model is worse on average; inside-llm tells you it is worse here, on this kind of token, at this confidence — and lets you check whether “here” is somewhere you care about. For anyone choosing quants for a local setup, that is a genuinely better question to be able to ask. If you run GGUF models locally, our hands-on with the llama.cpp 1.0 stable C API and the Colibri chat UI for GGUF models are the natural next stops.
García is asking the thread what to build next: which pairs to compare first — imatrix against non-imatrix, base against abliterated or fine-tuned, Unsloth UD against standard quants — and which per-position metric is still missing. Watch whether the 1.0 release, with 67 stars on GitHub at the time of writing, turns into a maintained tool or a weekend demo with a good video: the repository, the demo and the download packages are all live now, and the next release will say which one it is.
Sources: the original r/LocalLLaMA post by u/AlternativeEnough352; inside-llm on GitHub (manuelgscom/inside-llm); the inside-llm project page; the live in-browser demo. Upvote and comment counts as shown at the time of writing.