A 70-billion-parameter language model weighs about 140 GB in its original BF16 precision. That's more memory than even a flagship datacenter GPU like the NVIDIA H100 holds (80 GB). And yet, the same model runs on a single consumer GPU — or even a laptop CPU — at 4 bits per weight, at roughly 35 GB, and still writes coherent essays and debugs code.

The magic is called quantization: rounding a trained model's weights down from 16-bit floats to a compact number format. The surprise is that it works as well as it does — a 4× shrink with barely any quality loss. Here's how, and why naive rounding would fail where clever methods succeed.

Why memory, not compute, is the bottleneck#

Running an LLM is mostly a memory-bandwidth problem, not a compute problem. Each generated token requires streaming every weight through the processor once. In BF16, that's 2 bytes per parameter, so generating one token from a 70B model means reading 140 GB from memory. Cut the weights to 4 bits (half a byte) and you're reading ~35 GB instead — roughly 4× faster per token, and the model fits in hardware a quarter of the price.

The question is how much you can throw away. A weight stored in BF16 has about 4 decimal digits of precision. Round everything crudely and the model breaks: the quantization noise compounds through dozens of transformer layers until the output is garbage.

The core problem: outliers#

Naive quantization fails for a specific reason. In large models, a small number of outlier weights and activations have magnitudes tens to hundreds of times larger than the average. If you set your rounding grid wide enough to cover the outliers, the ordinary weights all round to nearly zero. If you set it to cover ordinary weights, the outliers saturate and get clipped — and those outliers turn out to be disproportionately important.

The landmark demonstration of this was LLM.int8() (Dettmers et al., 2022), which showed that protecting a small fraction of outlier weights at 16-bit while quantizing the rest to 8-bit avoided measurable quality loss all the way up to 175B-parameter models. The lesson stuck: every good quantization method is really an outlier-management strategy.

Method 1: GPTQ — repair the rounding error#

GPTQ (Frantar et al., 2023) is the classic 4-bit weight quantization method. It works layer by layer on the already-trained model — no retraining needed — using a small calibration dataset (typically a few thousand tokens of text).

The trick comes from classical model-compression math (Optimal Brain Surgeon). When you quantize one column of a weight matrix, you can compute exactly how much error that introduced, using the Hessian — a matrix describing how sensitive the layer's output is to each weight. GPTQ then nudges the not-yet-quantized weights in the opposite direction to cancel that error out. In plain language: it quantizes greedily, and fixes each mistake using information about which weights the layer can least afford to distort.

GPTQ typically works with a group size of 128 — each block of 128 consecutive weights shares a scale factor — which costs about 0.25 extra bits per weight on top of the 4, for roughly 4.25 effective bits. At 4-bit, GPTQ keeps perplexity within a few percent of the full-precision model on standard benchmarks; at 3-bit the quality falls off a cliff. Quantizing one 70B model with GPTQ is slow (hours on a good GPU), but you only do it once.

Method 2: AWQ — find the weights that matter#

AWQ (Lin et al., MIT, 2023) took a different route to the same destination. The authors observed that only about 1% of weight channels are "salient" — they interact with large-magnitude activations, and errors there propagate much further than errors elsewhere.

Before quantizing, AWQ scans a calibration dataset, finds the channels with large activation magnitudes, and applies a per-channel scaling that stretches the important weights so that more of the 4-bit grid is spent on them (mathematically dividing the activations by the same scale keeps the layer's output identical). It's like printing the fine print in a larger font before photocopying a page at low resolution.

On benchmark comparisons, 4-bit AWQ generally edges out GPTQ at the same bit-width — and it calibrates much faster, in tens of minutes rather than hours. For new deployments on NVIDIA hardware, AWQ is the default recommendation.

Method 3: NF4 — match the distribution#

If you're fine-tuning rather than just serving a model, you've likely met NF4, the 4-bit format introduced with QLoRA (Dettmers et al., 2023). NF4 stands for NormalFloat4, and its insight is about the shape of the grid, not just its resolution.

Four bits give you 16 representable levels. Uniform int4 spaces them evenly — say, -8 to 7. But neural-network weights are approximately normally distributed: most are near zero, a few are large. NF4 places its 16 levels at the quantiles of a standard normal distribution, so the levels are dense near zero (where most weights live) and sparse at the extremes. This is provably optimal for normally distributed data and measurably better than uniform 4-bit.

Practical NF4 details, as used in bitsandbytes:

  • Weights are normalized per block (typically 64 values) to [-1, 1], with a scale factor stored per block.
  • Double quantization quantizes the scale factors themselves to 8-bit, saving roughly 0.4 bits per weight with no practical quality impact.
  • Compute still happens in BF16: weights are dequantized on the fly for each matrix multiply, so numerical stability is preserved.

This is what makes QLoRA possible: a 65B model that needed over 780 GB of GPU memory to fine-tune the conventional way can be fine-tuned on a single 48 GB GPU.

The format you actually download: GGUF#

If you've run a local model through Ollama, LM Studio, or llama.cpp, you've downloaded a .gguf file — and the alphabet soup of Q4_K_M, Q5_K_M, Q8_0 inside it. One clarification: GGUF is a container format, not a quantization algorithm. It replaced the older GGML format in 2023 and bundles weights, tokenizer, and metadata in one portable file.

The quantizations inside GGUF files are block-based "K-quants." The default everyone reaches for, Q4_K_M, stores weights at roughly 4.8 bits each (extra precision for the most sensitive blocks, with scales folded in). A Llama-class 8B model lands around 4.9 GB at Q4KM; a 70B model around 48 GB. Community evaluations consistently show Q4KM holding onto the vast majority of full-precision quality, which is why Ollama ships most models at that level by default. Stepping up to Q5_K_M or Q8_0 buys progressively smaller gains for progressively more memory.

Where the losses actually land#

Quantization doesn't degrade a model uniformly. Common text generation holds up remarkably well at 4-bit. The tasks that suffer most are the ones requiring precise numerical reasoning, rare factual knowledge, or very long coherent chains — exactly the places where small rounding errors compound. This is why the honest advice is always the same: evaluate the quantized model on your actual workload, not on generic perplexity, and calibrate on domain-representative data when possible. A model quantized with medical-text calibration will treat medical text better than one calibrated on Wikipedia.

The direction of travel#

Weight-only 4-bit quantization was the state of the art in 2023. Since then the field has pushed in two directions. SmoothQuant (Xiao et al., 2022) migrates outlier difficulty from activations into weights, enabling true W8A8 quantization where activations are also 8-bit — faster integer arithmetic, same accuracy. Meanwhile, hardware is catching up: NVIDIA's Hopper GPUs added native FP8, and Blackwell's B200 supports native FP4, roughly doubling FP8 throughput on the same silicon.

The deeper story is that quantization keeps working because of something fundamental: trained networks carry a lot of redundancy. Four bits is enough not because 4 is a magic number, but because most of what a 16-bit weight says is noise the network never needed. The methods that win — GPTQ, AWQ, NF4 — are all ways of spending the limited bits where the signal actually is.

The takeaway#

  • Quantization attacks the memory wall: a 70B model drops from 140 GB (BF16) to ~35 GB (4-bit), which is what makes local and single-GPU inference possible.
  • Naive rounding fails because of outlier weights and activations; every good method is an outlier-management strategy.
  • GPTQ repairs rounding errors with second-order (Hessian) math. AWQ protects the ~1% of weight channels that matter most. NF4 reshapes the 4-bit grid to match the normal distribution of weights — and powers QLoRA fine-tuning on a single GPU.
  • GGUF is the container, not the algorithm; Q4_K_M (~4.8 bits/weight) is the community's default sweet spot.
  • Degradation is uneven: test on your own workload, and calibrate on representative data.

The bottom line: a 4-bit model isn't a damaged 16-bit model. It's a model with the noise squeezed out — usually indistinguishable from the original, until you ask it to do math.