The Race to Compress Intelligence - Quantization, Distillation, and the Future of Local AI
You've heard about GPT-4, Gemini, Claude, Llama. You've probably used one. But here's a number most people gloss over: a state-of-the-art large language model in full precision can require hundreds of gigabytes of memory just to load. That's before it processes a single word.
The Problem No One Talks About Enough
This creates a hard wall. Only companies with data centers full of expensive GPUs can serve these models. Everyone else is a paying customer, renting intelligence by the API call.
Quantization is one of the most important techniques trying to knock that wall down.
What Is Quantization?
Every large language model is, at its core, an enormous list of numbers called weights. These weights are the encoded knowledge of the model — everything it learned during training about language, reasoning, facts, and patterns. A "7 billion parameter" model has 7 billion of them.
The question quantization asks is deceptively simple: how precisely do we actually need to store each of those numbers?
By default, most models store weights in 32-bit floating point (float32). Each weight is a high-precision decimal — something like 0.473821947 — and costs 4 bytes of memory. For a 7B parameter model, that's roughly 28 GB of memory just to hold the weights.
Float16 halves that. Each weight uses 2 bytes instead of 4, bringing a 7B model to about 14 GB. This is what most "full precision" models on HuggingFace actually ship in.
Quantization takes this further. Instead of storing weights as floating-point decimals, it rounds them to lower-precision integers:
| Format | Bits per weight | 7B Model Size | Notes |
|---|---|---|---|
| float32 | 32 | ~28 GB | Full precision, rarely used for inference |
| float16 | 16 | ~14 GB | Standard baseline |
| int8 | 8 | ~7 GB | Good accuracy, widely supported |
| int4 | 4 | ~3.5 GB | The sweet spot for most local use |
| 1-bit | 1–1.58 | ~0.5 GB | Frontier research, covered below |
The savings are dramatic. A 7B model that required a high-end workstation at float32 fits comfortably on a consumer laptop at int4.
But here's the fundamental tension: the model learned its knowledge in high precision. You're now forcing that knowledge into a cruder representation. Some information gets lost in the rounding. The entire art of quantization is figuring out how to minimize that loss.
Before going deeper into how quantization works and where it fails, it helps to decode the filenames you'll actually encounter when downloading models.
Decoding the Numbers: What Q4_K_M Actually Means
If you've browsed Ollama or HuggingFace, you've seen model names like:
llama3-8b-Q4_K_M
mistral-7b-Q5_K_S
phi3-Q8_0
deepseek-Q2_KThis is the GGUF naming convention, the dominant format for locally-run quantized models. Once you know the system, it reads like a spec sheet.
The Q prefix means quantized.
The number (Q2, Q4, Q5, Q8) is the bits per weight. Q4 = 4 bits per weight. Q8 = 8 bits.
The K means the model uses K-quants — a more sophisticated quantization method that uses mixed precision. Different layers of the model get different levels of precision based on their importance, rather than compressing everything uniformly to the same bit depth.
The suffix (S, M, L) means Small, Medium, Large — how aggressively the K-quant method trades quality for size within that bit level:
_S— smaller file, slightly more quality loss_M— the balanced standard choice_L— larger file, better quality retention
The _0 suffix (as in Q8_0) means a simpler, legacy quantization scheme — no K-quants, uniform precision across all layers.
IQ variants (like IQ3_XXS) are importance-aware quants — a more advanced method that analyzes which weights matter most to model output and protects them with higher precision. These often outperform K-quants at the same average bit depth.
Rule of thumb: Q4_K_M or Q5_K_M is the right choice for most local deployments. Q8 is nearly lossless and worth it if VRAM allows. Below Q3, quality degradation becomes noticeable on most tasks.
How Quantization Actually Works
At its core, quantization is a mapping problem: take a value from a continuous high-precision range and map it to the nearest value in a discrete low-precision set.
For int8 symmetric quantization, the mapping is:
q = round( x / s )Where:
xis the original float32 valuesis the scale factor:s = max(|x|) / 127qis the resulting int8 value, clamped to [-127, 127]
To reconstruct the original value (dequantize):
x̂ = q × sThe reconstruction x̂ is not identical to x. The difference x - x̂ is the quantization error. For a single weight it's small. Across billions of weights, it accumulates — and in sensitive layers, it compounds through the forward pass.
Asymmetric quantization adds a zero-point z to handle distributions that aren't centered around zero:
q = round( x / s ) + z
x̂ = (q - z) × sThis matters because weight distributions in real models are often skewed. Forcing a symmetric scale wastes representable values on one side of zero.
The Outlier Problem
Quantization fails in two distinct ways. The first is magnitude distortion — the scale factor is dominated by outliers, destroying precision for small values. The second is information collapse — small values lose their signs entirely, becoming indistinguishable zeros. Both are caused by the same root issue, but they damage the model differently.
The scale factor s is determined by the maximum absolute value in the tensor.
Consider a weight tensor of 128 values where 126 are in the range [-0.5, 0.5] and two outliers sit at +14.2 and -11.8. The scale factor becomes:
s = 14.2 / 127 ≈ 0.1118Now quantize a small weight, say x = 0.08:
q = round(0.08 / 0.1118) = round(0.716) = 1
x̂ = 1 × 0.1118 = 0.1118The error is 0.08 - 0.1118 = -0.0318 — a 39% relative error on that weight. Meanwhile the outlier:
q = round(14.2 / 0.1118) = round(127.0) = 127
x̂ = 127 × 0.1118 = 14.2Reconstructs almost perfectly. The outlier dominates the scale; the small weights absorb the error.
At 4-bit this is more severe. Only 16 representable values (-8 to +7):
s = 14.2 / 7 ≈ 2.028
q = round(0.08 / 2.028) = round(0.039) = 0That small weight quantizes to zero and is completely lost.
Sign Loss: The Subtler Failure
The outlier problem destroys magnitude precision. But there is a second failure mode that is arguably worse: sign loss.
In the example above, x = 0.08 quantized to zero. Now consider two weights: x₁ = +0.08 and x₂ = -0.08. Both round to zero. They are now indistinguishable.
This matters because sign encodes semantic direction in the vector space. In attention mechanisms, the dot product between a query vector and a key vector determines how much attention a token pays to another:
attention_score = Σ (q_i × k_i)Each term contributes positively or negatively based on the signs of q_i and k_i. If small values lose their signs — collapsing to zero — those contributions vanish entirely. The attention score is no longer computing what the model learned to compute. It's computing a degraded approximation that systematically ignores the subtle alignment signals between tokens.
At float16 you can represent values as small as ~6×10⁻⁵ with correct sign. At Q4 with an outlier-blown scale of 2.028, anything with absolute value below ~1.0 collapses to zero. You are not just losing precision — you are losing the sign of a large fraction of the vector's components.
How Block Quantization Restores Sign
The fix is local scale computation. Instead of one scale factor for the entire tensor, block quantization divides the tensor into small blocks — typically 32 values — and computes an independent scale per block:
s_block = max(|x_i| for i in block) / (2^(bits-1) - 1)The two outliers (+14.2, -11.8) land in one block. Their block scale is large — but it only governs those 32 values. Every other block computes its own tight scale based on its actual range.
For a block containing only small weights in [-0.5, 0.5]:
s_block = 0.5 / 7 ≈ 0.0714 (4-bit)Now quantize x₁ = +0.08 and x₂ = -0.08:
q₁ = round(+0.08 / 0.0714) = round(+1.12) = +1
q₂ = round(-0.08 / 0.0714) = round(-1.12) = -1Signs are preserved. Values are distinguishable. The attention mechanism gets the directional signal it needs.
Block quantization is not just a size optimization — it is a correctness fix. Without it, the model is not just degraded — it can become functionally unreliable on tasks that depend on fine-grained attention signals.
K-Quants: Mixed Precision Across Layers
Block quantization solves the local outlier problem. K-quants extend this further by recognizing that not all layers of a model are equally sensitive to quantization error.
Attention layers and certain projection matrices are more sensitive than feed-forward layers. K-quants assign higher bit depth (6-bit or 8-bit) to sensitive layers and lower bit depth (3-bit or 4-bit) to less sensitive ones — targeting a specific average bit depth across the model while protecting what matters most.
This is why Q4_K_M at an average of 4 bits per weight consistently outperforms a naive uniform Q4_0 despite nominally similar compression ratios.
Post-Training Quantization vs. Quantization-Aware Training
There are two fundamentally different moments when quantization can happen:
Post-Training Quantization (PTQ) — the model is trained normally in full precision, then quantized afterward as a separate step. Every GGUF file you download is PTQ. Fast and cheap, requires no retraining, but the model was never prepared for the precision loss.
Quantization-Aware Training (QAT) — quantization is simulated during training. The model learns to be robust to lower precision from the start, because the rounding is part of the training signal. QAT models generally perform better at the same bit depth, but require training from scratch — expensive and time-consuming. This is the approach behind BitNet, covered below.
Where Error Accumulates
Quantization error is not uniformly distributed. It is worst in:
- Early layers — errors propagate and amplify through every subsequent layer
- Attention mechanisms — sensitive to small value differences in key/query dot products
- Embedding and output projection layers — directly affect token probability distributions
Methods like GPTQ, AWQ, and IQ-quants focus disproportionate effort on protecting these layers. GPTQ uses second-order gradient information (the Hessian) to find the optimal rounding direction for each weight — minimizing the cumulative output error rather than just the per-weight error. AWQ identifies the 1% of weights that are most activation-sensitive and preserves them at higher precision. Both outperform naive PTQ at the same bit depth, at the cost of more compute during the quantization step itself.
The KV Cache: A Different Quantization Problem
Everything above addresses weight quantization — compressing the stored knowledge of the model. But there is a second, separate memory problem that only appears at inference time: the KV cache.
When a model processes your prompt and generates a response, it needs to reference every previous token to compute attention. This memory is stored in the Key-Value cache: for every token in the context, the model stores two vectors (a Key and a Value), each typically 128 dimensions long.
The memory cost scales as:
KV cache size = 2 × d_head × n_heads × n_layers × context_length × bytes_per_valueFor a model with a 128K context window running at float16, this can reach gigabytes of KV cache alone — separate from the model weights. Extend the context, and the cache grows linearly. This is why long contexts are expensive and why consumer hardware hits a wall well before the advertised context limit becomes practical.
KV cache quantization compresses these cached vectors from float16 to 4-bit, reducing cache memory by approximately 4×. In theory, this means fitting a context window 4× larger on the same hardware.
But KV cache quantization hits the outlier and sign loss problems described above — and often harder, because activation values at inference time have more extreme outlier distributions than static weights. Naive 4-bit compression of the KV cache degrades model output quality noticeably.
TurboQuant, RotorQuant, and IsoQuant
TurboQuant: Rotation Before Compression
Google's TurboQuant addresses the outlier problem with a mathematically elegant solution: apply a rotation matrix to the vector before quantizing, then reverse the rotation after dequantizing.
The rotation does not change the information content of the vector — it is a lossless linear transformation. But it redistributes the energy of outlier values across all dimensions. A vector with two extreme values and 126 near-zero values becomes a vector where all 128 values are of roughly similar magnitude. The quantization scale is no longer dominated by outliers. Small values — and their signs — survive compression.
The rotation matrix R is a fixed 128×128 orthogonal matrix. The quantization pipeline becomes:
store: q = quantize( R × v )
retrieve: v̂ = R⁻¹ × dequantize( q )Since R is orthogonal, R⁻¹ = Rᵀ — cheap to compute.
The result: KV cache compressed to 4-bit with quality comparable to float16, achieving approximately 5× memory reduction while preserving context accuracy.
The cost: the 128×128 matrix multiplication requires 16,384 multiply-add operations per vector. Scaled across an entire context:
16,384 ops × 2 (K + V) × context_length × n_heads × n_layersFor a long context, this adds billions of additional operations to the pre-fill phase — the step where the model processes your input prompt. Pre-fill latency, already slow for long contexts, gets significantly worse.
RotorQuant and IsoQuant: Geometric Algebra Approximations
The open-source response was to ask: does the rotation need to be a full 128×128 matrix?
The answer is no — not if the goal is simply to redistribute outlier energy, rather than achieve a mathematically perfect rotation. Techniques borrowed from 3D graphics provide much cheaper approximations.
RotorQuant splits the vector into groups of 3 elements and applies geometric algebra rotors to each group independently. Each group of 3 requires far fewer operations than a full matrix multiply.
IsoQuant refines this further by splitting into groups of 4 elements — 32 groups for a 128-dimension vector, with no remainder. Each group is transformed by a quaternion: a 4-component rotation representation standard in 3D game engines.
The arithmetic per vector:
| Method | Operations per vector | Data movement |
|---|---|---|
| TurboQuant (dense matrix) | 16,384 | 128×128 matrix from VRAM |
| IsoQuant (quaternions) | 512 (16 ops × 32 groups) | 512 bytes, fits in GPU registers |
That is 32× less compute and 128× less data movement. Theoretical speedups suggest up to 19× on Nvidia and 31× on Apple Silicon over TurboQuant's rotation step.
An important caveat: TurboQuant likely does not use a naive dense matrix multiply in practice. The real-world compute gap between TurboQuant and IsoQuant may be significantly narrower than the theoretical numbers suggest. The benchmarks, once IsoQuant's GPU kernels are fully implemented, will tell the real story.
Implementation status at time of writing: IsoQuant and PlanarQuant exist in a community llama.cpp fork. On Apple Silicon, testing revealed 34 graph splits per inference pass instead of the expected 2 — indicating the Metal GPU kernels were not yet implemented and work was falling back to CPU. The theoretical gains are real; the production implementation is still maturing. TurboQuant already has full kernel implementations and delivers clean inference today.
Distillation: A Fundamentally Different Approach
Quantization compresses an existing model. Distillation creates a new, smaller model by teaching it to mimic a larger one.
The setup: a large, highly capable teacher model trains a smaller student model — not on raw ground-truth data, but on the teacher's output distributions. The student learns to reproduce not just the teacher's answers, but its full probability distribution over possible next tokens. This is richer training signal than ground truth alone: the distribution encodes the teacher's uncertainty, its sense of which alternatives are plausible, and the relative confidence across the vocabulary.
The loss function for distillation is typically a combination of:
L = α × L_CE(student, ground_truth) + (1-α) × L_KL(student, teacher)Where L_KL is the KL divergence between the student and teacher distributions, and α balances between fitting the data and mimicking the teacher.
This is why distilled models often outperform models of equivalent size trained from scratch. A small model trained from scratch only sees binary supervision — correct or incorrect. A distilled student sees the teacher's full reasoning expressed as a probability landscape.
Real examples: DeepSeek-R1 distilled into 7B and 14B models that outperform same-size models on reasoning benchmarks. Many of the competitive small models available today are distilled from larger ones, then quantized for deployment.
Quantization vs. Distillation
| Quantization | Distillation | |
|---|---|---|
| What it does | Compresses an existing model | Creates a new, smaller model |
| Starting point | A trained model | A trained teacher + full training run |
| Compute cost | Low (post-training) | High (requires retraining) |
| Architecture | Identical to original | Can differ from teacher |
| Quality ceiling | Bounded by original model | Can exceed original on specific tasks |
They are complementary, not competing. A distilled model can be quantized. Most of the best small models you can run locally today are both — distilled from a large teacher, then quantized for deployment.
BitNet: The 1-Bit Frontier
Everything discussed so far compresses models that were trained in full precision. BitNet is different: a model trained from scratch to use only ternary weights.
The Core Idea
In BitNet b1.58, each weight can only be one of three values: {-1, 0, +1}. This is technically 1.58 bits (log₂(3)). The model must be trained with these constraints from the beginning — it cannot be derived from an existing full-precision model by post-training quantization.
The key finding from Microsoft Research's BitNet paper: at sufficient scale, the scaling laws still hold. Bigger BitNet models are more capable, just as with full-precision models. The model learns to encode useful representations within the ternary constraint, rather than the constraint preventing learning.
The open question is not whether 1-bit models scale — it is whether they can preserve the fine-grained reasoning behavior that higher-precision models exhibit, especially on long-chain or tool-augmented tasks. Benchmark scaling is one thing; reasoning fidelity under complex, multi-step prompts is another.
Hardware Implications
Multiplying by {-1, 0, +1} requires no floating-point multiplier. It is addition, subtraction, or a no-op. Modern chips spend significant silicon area and power budget on floating-point units. Hardware designed specifically for BitNet inference could be dramatically simpler, cheaper, and more power-efficient than current GPU architectures.
The memory reduction is approximately 14× smaller than the equivalent float16 model at the same parameter count.
Where Things Stand
Microsoft's published BitNet models (2B, 3B, 8B) were trained on limited data and are not competitive with full-precision models of the same size in practice.
The first commercially viable 1-bit models came from Prism ML, whose Bonsai series achieved quality comparable to full-precision counterparts at the same parameter count. The 8B Bonsai model runs at approximately 130 tokens/second on consumer hardware — competitive with quantized full-precision models — at a fraction of the memory footprint.
The constraint is the training cost. You cannot convert an existing Llama or Gemma model to BitNet. It requires training from scratch — an investment of hundreds of thousands of dollars in compute. The open-source community has not yet fully committed because the economics remain uncertain relative to aggressive PTQ of full-precision models.
The Combination
These techniques are not mutually exclusive. 1-bit weights reduce the model's memory footprint at rest. KV cache quantization (TurboQuant/IsoQuant) reduces the memory required for long contexts at inference time. Together, they point toward models that are both small enough to deploy on consumer hardware and efficient enough to handle long contexts on that same hardware — without the memory pressure that currently makes long-context inference impractical outside of data centers.
Why This Matters
All three techniques attack the same constraint from different angles: quantization compresses representation, distillation compresses behavior, and 1-bit training redefines the representation entirely. Together they form a compounding force — each advance making the next more practical.
The standard framing for AI progress is capability — what can the model do? But capability without accessibility is a product that serves only those with the infrastructure to run it.
The question is no longer just how powerful models become. It is who can afford to run them — and who cannot.
Quantization, distillation, and 1-bit training are the mechanisms shifting that answer. The trajectory is consistent: techniques that required a data center five years ago run on a laptop today. Understanding where they work, where they fail, and what tradeoffs they make is not academic. It is understanding the actual engineering constraints that determine who gets access to capable AI, and when.
References: Microsoft Research BitNet b1.58 (2024); Google TurboQuant; RotorQuant / IsoQuant llama.cpp fork (SCREA); Prism ML Bonsai; GPTQ (Frantar et al., 2022); AWQ (Lin et al., 2023).
Shaped in collaboration with Claude, an AI assistant by Anthropic, on a bright spring afternoon — the kind where sunlight makes you optimistic enough to believe a trillion parameters might actually fit in your pocket.