← All writing
10 min readAISystemsProductionLLMsInference

LLM.int8(): what the 8-bit matrix multiplication paper actually says

Reading Dettmers, Lewis, Belkada, and Zettlemoyer (UW/HuggingFace, NeurIPS 2022) after a quantized model passed all our evals and then failed in production.

The model was OPT-30B. We needed it on a single A100 80GB for a latency-sensitive deployment — at FP16, OPT-30B is 60 GB of weights, leaving almost nothing for KV cache and activations. The obvious move: quantize to INT8. Half the memory, same model.

We used a standard post-training INT8 quantization library — the kind that works without issues on ResNet-50, EfficientNet, and CLIP image encoders. Perplexity on Wikitext-2: 10.7 in FP16, 11.1 in INT8. Within 4% — acceptable. We shipped.

The failure showed up in code generation tasks. The model would produce plausible-looking Python with subtle semantic errors that were invisible to perplexity: variable names that didn't match, off-by-one errors in loop bounds, function arguments in the wrong order. The quantized model was systematically less reliable on tasks involving precise token-level reasoning, without any perplexity signal that anything was wrong.

Investigating this is what led me to the LLM.int8() paper — "LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale," Dettmers, Lewis, Belkada, and Zettlemoyer, NeurIPS 2022. The paper explains exactly why standard INT8 quantization fails for large LLMs and what to do instead. The mechanism isn't a subtle accuracy bug — it's a phase transition in how transformers represent information that appears at a specific scale.

The problem the paper is actually solving

To understand why INT8 fails for LLMs, you need to understand what INT8 quantization actually does.

Standard INT8 quantization maps a floating-point tensor to signed 8-bit integers (−128 to 127). The mapping: find the max absolute value in the tensor, set that as the "scale," and divide every value by the scale. You get integers in [−127, 127], stored as INT8. At compute time, you do integer matrix multiplication, then rescale the output back to float.

The problem is that this mapping is determined by the outliers. If your weight tensor has values uniformly distributed in [−0.5, 0.5], the scale is 0.5 and you have 254 integer values spanning the whole range — excellent resolution. If the same tensor has one entry at 150.0 and everything else in [−0.5, 0.5], the scale is 150.0. Now your 0.5 values quantize to integer 0 or 1 (out of 127). You've effectively rounded all the moderate-magnitude values to zero or near-zero.

For vision models, this is a known and manageable problem. Outliers exist but are rare and bounded. The standard fix — per-channel quantization, where each channel gets its own scale — handles the variance without much accuracy loss.

The paper's core finding is that LLMs develop a qualitatively different problem at scale: emergent outlier features that violate the assumptions that make standard INT8 quantization tractable.

Outlier features: what they are and when they emerge

The paper defines outlier features as hidden state dimensions with magnitude significantly larger than the rest of the tensor. Not 2× or 5× larger — 100× to 1000× larger.

These aren't random noise. They're systematic:

  • The same specific dimensions are outliers across all token positions for a given layer
  • The same dimensions are outliers across all input examples — different prompts, different tasks
  • Once a dimension becomes an outlier in a layer, it stays an outlier for the lifetime of the model

This makes them fundamentally different from the occasional outliers you see in vision models, which are localized and input-dependent. LLM outlier features are structural properties of the model's learned representation.

The paper plots outlier emergence across model scales using the OPT family (125M to 175B):

  • OPT-125M, OPT-350M: essentially no outliers
  • OPT-1.3B: occasional outliers in some layers
  • OPT-6.7B: most layers have outliers; ~75% of layers affected
  • OPT-13B and above: essentially all layers have outliers; the feature is ubiquitous

The transition is sharp. Below 1B parameters, standard INT8 quantization works without any special handling. Above 6.7B, it doesn't. This explains why INT8 was considered a solved problem for a long time — it was, for the model sizes people were deploying before LLMs.

Why do outlier features emerge? The paper doesn't claim a definitive answer, but the evidence suggests they're functional: attention mechanisms use specific dimensions to store information that needs to be reliably attended to or suppressed across the entire sequence. The model learns to concentrate certain "global" computations into dedicated dimensions with high dynamic range. This is related to (but distinct from) the attention sink phenomenon — the paper cites that large attention weights on specific tokens also correlate with large hidden state magnitudes at those positions.

What happens when you quantize a tensor with outliers

Concretely: take a hidden state vector from OPT-30B at FP16. At layer 48, dimension 892 might have value 147.3. The remaining 4,095 dimensions have values in the range [−0.8, 0.8].

Per-tensor INT8 quantization sets scale = 147.3 / 127 ≈ 1.16. Every other dimension quantizes to:

round(0.4 / 1.16) = round(0.345) = 0   → dequantized: 0.0    (error: 0.4)
round(0.8 / 1.16) = round(0.690) = 1   → dequantized: 1.16   (error: 0.36)
round(-0.3 / 1.16) = round(-0.258) = 0 → dequantized: 0.0    (error: 0.3)

The quantization error for the normal-magnitude values is comparable to the values themselves. You haven't approximated the hidden states — you've replaced most of them with zero or with ±1.16.

Per-channel quantization helps (you'd quantize dimension 892 separately from dimension 400), but the problem propagates through matrix multiplication. When you multiply a quantized activation matrix by a weight matrix, the quantization error in one layer feeds into the next. With weights also quantized, the compounding errors across 96 layers of a 30B model accumulate into significant quality degradation.

The paper's perplexity numbers tell part of the story. For OPT-175B with standard INT8 quantization: perplexity degrades from 8.34 to 23.1. That's not a minor accuracy hit — the model has collapsed. For OPT-125M: FP16 perplexity 27.6, INT8 perplexity 27.8. The small model is fine.

The subtlety: perplexity on clean text doesn't always capture the degradation on structured tasks. Our code generation failure was a case where the model maintained "plausible text" distribution but lost precise token-level coherence — the kind of local reasoning that depends on the high-precision activations the outlier dimensions carry.

The fix: mixed-precision decomposition

The insight driving LLM.int8() is that if outlier features are systematic, they're also predictable at inference time. You can identify which dimensions are outliers and handle them differently.

The algorithm for a single matrix multiplication Y = X @ W (where X is the activation matrix and W is the weight matrix):

Step 1: Detect outlier dimensions. At each forward pass, check which columns of X have magnitude above a threshold (the paper uses 6.0 as default, tunable per model). Call the outlier column indices O and the normal column indices N.

Step 2: Partition. Split X into X_out (columns in O) and X_norm (columns in N). Split W into W_out (corresponding rows in O) and W_norm (remaining rows in N).

Step 3: Compute with appropriate precision.

Y_out  = X_out  @ W_out    # computed in FP16 — preserves outlier precision
Y_norm = X_norm @ W_norm   # computed in INT8 — fast, correct for normal values

Step 4: Recombine.

Y = Y_out + dequantize(Y_norm)

The key quantities: |O| / total_dims ≈ 0.1% for most layers. You're doing FP16 multiplication for a 0.1% slice and INT8 for the remaining 99.9%. The FP16 portion is small enough that it doesn't dominate computation time. The INT8 portion is large enough to give you most of the memory and bandwidth savings.

The bitsandbytes library (by Dettmers, the first author) implements this. In Hugging Face transformers:

from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained(
    "meta-llama/Llama-2-70b",
    load_in_8bit=True,
    device_map="auto",
)

Under the hood, each nn.Linear layer is replaced with a Linear8bitLt module that performs the mixed-precision decomposition on every forward pass. The weight matrix is stored in INT8; the outlier detection and FP16 path happen at runtime.

What the paper measures

The results focus on three model families: OPT (125M to 175B), BLOOM (560M to 176B), and EleutherAI's GPT-J/NeoX models.

Key findings:

Quality preservation. For models ≥ 6.7B, INT8 with mixed-precision decomposition matches FP16 perplexity within 0.1 points on Wikitext-2, C4, and PTB. For OPT-175B: FP16 perplexity 8.34, LLM.int8() perplexity 8.35. Standard INT8 got 23.1. The decomposition fully recovers quality.

Zero-shot task performance. Across Winogrande, HellaSwag, PiQA, BoolQ, WinoGrande: LLM.int8() matches FP16 within noise for all model sizes ≥ 6.7B. The code generation failure we saw with naive INT8 wouldn't happen with LLM.int8().

Memory savings. OPT-175B: 340 GB (FP16) → ~180 GB (INT8). A model that required more than two 8×A100 servers fits on one. BLOOM-176B: same story. This was the practical unlock — it made 100B+ models accessible to teams without petascale GPU clusters.

Throughput tradeoffs. Here the picture is more nuanced, and the paper is honest about it. For batch size 1, LLM.int8() is 15–23% slower than FP16 on A100. The decomposition has overhead — splitting the matrix, running two separate matmuls, recombining. At batch size 1, you're not compute-bound, and the overhead hurts.

The throughput advantage appears at larger batch sizes. At batch 8+, the reduced memory footprint lets you run more requests concurrently; the actual INT8 matmul is faster; and the amortized overhead of the decomposition matters less. For throughput-oriented serving (high batch sizes, offline processing), LLM.int8() improves overall system throughput compared to FP16 on the same hardware.

When NOT to use LLM.int8()

Given the GPTQ and AWQ posts on this blog, it's worth being precise about where LLM.int8() belongs versus alternatives.

When you're memory-bandwidth-bound at batch size 1. Most LLM decode inference at low request volume is memory-bandwidth-bound — you're loading weights from HBM faster than you can compute with them. INT8 weights load twice as fast as FP16 weights, which should help. But the mixed-precision decomposition adds FP16 operations and synchronization overhead that can negate this. Measured decode speed is often comparable between LLM.int8() and FP16 at batch 1. If your goal is minimum latency for a single user, LLM.int8() may not help.

When you have access to H100 FP8. H100 hardware supports FP8 matrix multiplication natively with no software decomposition overhead. FP8 gives similar memory savings to INT8 with better accuracy (FP8 retains the floating-point dynamic range that makes outliers less catastrophic) and better performance. If you have H100s, use FP8 via PyTorch AMP or TensorRT-LLM rather than LLM.int8().

When you need INT4 compression. LLM.int8() gives ~2× memory compression. For many deployments, you need 4×. GPTQ or AWQ at INT4 give that. These approaches calibrate quantization to minimize layer-wise error rather than decomposing outliers — they solve the outlier problem differently and achieve better compression at the cost of a one-time calibration pass. If your constraint is memory and you can tolerate calibration time, INT4 is usually the right target.

When your model is under 1B parameters. The outlier phenomenon doesn't exist at small scale. Standard INT8 quantization without the mixed-precision overhead will work and will be faster. LLM.int8()'s decomposition adds latency for no quality benefit at small scales.

When you're doing fine-tuning. LLM.int8() is an inference-time technique. For training, you want QLoRA (4-bit NormalFloat quantization of the base model + LoRA adapters in FP16). The bitsandbytes library supports this too, but through a different codepath — load_in_4bit with bnb_4bit_compute_dtype=torch.bfloat16.

The failure modes that matter in production

Threshold sensitivity. The outlier detection threshold (6.0 by default) isn't universally correct. Some model families have outliers at magnitudes above or below this. Symptoms of wrong threshold: quality is worse than expected but not catastrophically bad. Diagnosis: profile the activation magnitudes in a representative forward pass and check if your threshold is capturing the actual outlier distribution. bitsandbytes lets you set this via llm_int8_threshold.

Embedding layers aren't quantized. The input embeddings and LM head are kept in FP16. For LLaMA-7B, the embedding table is ~32K vocab × 4096 dim × 2 bytes ≈ 262 MB. Not significant. For models with very large vocabularies (100K+), the unquantized embeddings become a non-trivial fraction of total memory.

Not all matrix types benefit equally. Attention QKV projections, FFN layers, and output projections all get quantized. The actual attention score computation (QK^T, softmax, AV) is not affected — that's between activations, not between activations and weights. If your memory bottleneck is KV cache rather than weights, LLM.int8() doesn't help.

Hardware requirements. INT8 matrix multiplication requires CUDA compute capability sm_75 or higher (Turing architecture: T4, RTX 2080, RTX 3090, A10, A100). Older GPUs (P100, V100) don't have the hardware INT8 matmul units. You'll get an error or fall back to FP16 at runtime.

Quantization is per-model, not per-request. The INT8 weights are fixed at load time. You can't dynamically adjust precision for specific requests. If you have a mixed workload where some queries need high precision and others don't, you're stuck serving everything at the same quantization level.

Why this paper mattered

Before LLM.int8(), the practical constraint for serving large open-weight models was that 30B+ parameter models required multi-GPU setups that most teams couldn't afford for a single serving replica. OPT-175B required at least 5 A100-80GB GPUs in tensor parallel. BLOOM-176B the same.

LLM.int8() (and the bitsandbytes library) changed that. Within weeks of the paper dropping, the open-source community was running 30B models on single A100s and testing 65B models on 2-GPU setups. It's the reason the phrase "run LLaMA-65B on 2 A100s" was technically accurate in late 2022 — the INT8 decomposition made it possible.

More importantly, the paper identified the outlier feature phenomenon as a fundamental property of scale, not an artifact of any specific architecture. This finding directly influenced GPTQ (which has to handle outliers during calibration) and AWQ (which finds the 1% of weights most sensitive to quantization, a complementary insight to LLM.int8()'s focus on 0.1% of activations with extreme magnitude). Understanding why INT8 fails for LLMs at scale is prerequisite knowledge for reasoning about why any quantization approach works or doesn't.

The lesson that didn't make it into the paper title: perplexity is an insufficient proxy for quality on structured tasks. Our production failure had perplexity within 4% of FP16 and was reliably wrong on code generation. Standard INT8 quantization was compressing the token distributions in ways that looked fine on next-token prediction but broke the precise conditional reasoning that code requires. Measuring what matters for your actual use case — not perplexity — would have caught this before deployment. LLM.int8() solves the quantization problem. Knowing what to measure is still your job.