StreamingLLM: what the attention sink paper actually says

Reading Xiao et al. (MIT/Meta, 2023) while debugging a long-running chat assistant that started producing incoherent output after 2,000 tokens.

The assistant worked fine for the first thousand tokens. Then around token 1,800, it started producing repetitive filler. By token 2,400, it had lost track of what it was supposed to be doing entirely. We were using a 4K context model, so we'd implemented a simple sliding window — evict the oldest tokens when the cache fills, keep the most recent. That made sense.

That's also what broke it.

Evicting the first tokens was the failure. We tried keeping the system prompt around and sliding only the user turns, and things improved. But we didn't know why, or what the principled fix was, until I read "Efficient Streaming Language Models with Attention Sinks" by Xiao, Tian, Chen, Han, and Lewis (MIT/Meta, 2023).

The answer involves a property of transformer attention I hadn't thought carefully about: the model is using the first tokens as a dumping ground for attention probability mass. Evicting them is catastrophic for the same reason evicting a garbage collector from memory would crash a runtime.

The problem the paper is actually solving

Language models have a fixed context window. For tasks that run longer than the window, you need a strategy. Three broad options:

1. Full context: recompute everything every step. Memory and compute scale with O(N²). Infeasible for very long sequences.
2. Window attention: maintain a sliding window of the last K tokens. Memory is O(K). But evicting old tokens causes a position mismatch — the KV cache has entries at positions 1..K, but after eviction, position encodings are stale.
3. StreamingLLM: sliding window plus permanently-kept "sink" tokens at the start. Memory is O(sink_size + window_size). Perplexity stays stable.

The paper benchmarks all three. The key finding is that window attention doesn't gradually degrade — it collapses. Perplexity shoots up catastrophically as soon as the initial tokens are evicted. Something about those first tokens is essential, independent of their content.

Attention sinks: what they are and why they form

Look at the attention maps of a trained transformer. The first few tokens — typically the initial BOS token, the first sentence, the first few words — receive disproportionately high attention from nearly every head across all layers, regardless of what question is being asked. A query at token 1,800 pays attention to the opening BOS token almost as much as to recent context. This pattern is consistent across models: LLaMA, GPT-2, Falcon, Pythia. It's not a quirk of one architecture.

The cause is softmax. Transformer attention computes:

Attention(Q, K, V) = softmax(QK^T / sqrt(d)) · V

The softmax normalizes the weights to sum to 1. That means the model must put probability mass somewhere for every query. When no key is particularly relevant to the current query — which is common when the model is generating bridging or filler content — the weights still have to sum to 1. The model learns to solve this by creating dedicated receptacle tokens: positions that reliably absorb the excess mass.

The first tokens in the sequence are ideal for this role. They're always present (never evicted in normal operation), they've attended to every subsequent token during training (causal attention means early tokens are in every query's history), and the model can reliably learn to route attention mass to them without those tokens "fighting back" semantically. They become garbage collection sinks for probability mass that has nowhere meaningful to go.

This is an emergent optimization artifact. The model isn't doing anything unusual; it's solving the softmax-normalization constraint as efficiently as it can. The first tokens become structurally important not because of their content, but because of their persistent position in every sequence.

The StreamingLLM algorithm

The fix is simple once you understand the sink mechanism:

Keep the first s tokens in the KV cache permanently. Maintain a sliding window of the last w tokens. Total KV cache size: s + w.

The cache looks like:

[ sink_0 | sink_1 | ... | sink_s-1 | ... recent_{w-k} | ... | recent_w ]

The sinks absorb the excess attention mass that would otherwise destabilize the softmax distribution. The window provides actual recent context. Together, they let the model generate arbitrarily long sequences with stable perplexity.

The paper tests s = 4 sink tokens across a range of window sizes. With a 4K window and 4 sink tokens, StreamingLLM maintains stable perplexity across 20K+ tokens where window-only attention fails after 4K. Quality plateaus at roughly the level of a 4K-context model — you're not getting true long-range recall, but you're not getting collapse either.

Cache management requires a position remapping step. When you evict tokens from the middle of the KV cache (old window entries), the remaining entries have stale position indices. StreamingLLM handles this by recomputing RoPE position embeddings at attention time using the token's current cache position rather than its original sequence position. Sink tokens stay at positions 0..s-1; window tokens shift down as entries are evicted. The model sees a consistent, recency-ordered context with stable positional information.

The trained-sink variant

The paper also introduces a cleaner approach that requires fine-tuning: replace the accidental BOS-as-sink with an explicit dedicated sink token.

You add a new special token — <sink> — at the start of every sequence during training. The model learns to use this token as its attention garbage collector from the beginning. At inference time, you keep exactly one dedicated sink token in the KV cache. Memory cost for sinks drops from 4 tokens to 1.

Trained-sink models show better streaming performance than those relying on natural sink formation. The model doesn't have to discover that the initial tokens are supposed to absorb excess attention; it has a designated place to put it from the first gradient update.

From a systems standpoint, the trained-sink approach also simplifies cache management: position 0 is always the sink, positions 1..w are the recent window. No ambiguity about which initial tokens to preserve.

Production tradeoffs

Memory. For a model with 32 attention heads and 128-dimensional keys/values in BF16, a KV cache entry costs 2 × 32 × 128 × 2 = 16 KB per token. A 4K window with 4 sinks uses 4,100 × 16 KB ≈ 63 MB per sequence — fixed, regardless of how long the conversation runs. Standard attention on a 100K-token conversation needs 100,000 × 16 KB = 1.5 GB per sequence. For a system serving thousands of concurrent long-running sessions, the memory savings are the whole argument.

Quality ceiling. StreamingLLM maintains stable local coherence. It cannot maintain long-range coherence — information from 6K tokens ago is gone if your window is 4K. This is acceptable for many tasks (conversational assistants, streaming document generation) and catastrophic for others (technical support where the user's problem was described 8K tokens ago, code generation where the function signature appeared at context start).

Latency. Because the KV cache is bounded, prefill costs for new tokens are bounded. Standard full-context models have prefill time that grows quadratically with session length — a problem that compounds over long sessions even when you're not streaming. StreamingLLM's prefill cost is constant after the window fills.

In-window decode quality. The 4-sink design performs on par with full attention for sequences within the window length. The paper shows perplexity on the in-window portion of long documents is nearly identical to a model with full attention over the same window. The sinks have minimal semantic effect on generation quality for in-scope tokens; they're absorbing probability mass, not injecting noise.

Throughput. No changes to the attention kernel are required — you're just changing which KV entries are kept. This makes StreamingLLM straightforward to implement on top of existing inference infrastructure. PagedAttention or vLLM can manage the sliding cache with minimal modification.

What the paper doesn't solve

True long-range recall. If you need to refer to document section from 20K tokens ago, StreamingLLM gives you nothing. The KV cache has evicted it. For this you need retrieval (RAG), explicit summarization, or a model trained for long contexts with architectural changes (Mistral's sliding window attention, ALiBi, or simply fine-tuning on very long sequences).

Attention sink location variance. The paper characterizes sink behavior in standard pretrained models. Not all models form sinks at the same positions or with the same intensity. The 4-token heuristic works well empirically but isn't formally derived. Models with different position encoding schemes, unusual special token configurations, or that use prefix-locking during training may require different sink budgets.

Multi-turn important context loss. Long-running sessions accumulate critical state in the early context: user preferences, problem descriptions, prior decisions. StreamingLLM evicts this alongside everything else beyond the window. The practical fix is hybrid: combine StreamingLLM with a separate state store that re-injects compressed session summaries as refreshed "sink" content. This adds latency and implementation complexity that the paper doesn't address.

When NOT to use this

Don't use StreamingLLM when your task genuinely needs full context. Legal document analysis, codebase-wide refactors, long-form research synthesis — if relevant information could be anywhere in a 100K-token document and you need to synthesize across it, a sliding window with sinks gives you false confidence. You'll get coherent output that ignores 90% of the input without knowing it.

Don't use it when you can afford a long-context model. For modern inference, 128K-context models have largely solved the practical bounded-context problem for common use cases. StreamingLLM is most relevant for edge deployment, latency-critical systems where full KV cache materialization is too expensive, or serving older models where long-context fine-tuning isn't available.

Don't use it if you need exact recall of early context beyond the window. The system prompt, the initial task description, the user's initial constraints — these get evicted unless you explicitly pin them. If your application semantics require the model to accurately remember the beginning of the conversation at any point in an arbitrarily long session, StreamingLLM alone isn't sufficient. It needs to be paired with retrieval or context compression.

Don't use the pretrained-sink version when you control training. If you're fine-tuning a model for a streaming use case, train with a dedicated sink token from the start. The pretrained-sink behavior is reliable enough for serving existing models but leaves performance on the table compared to models explicitly optimized for it.

The thing worth keeping

The mechanism — emergent attention sinks from softmax normalization constraints — is more important than the specific algorithm.

Transformers learn to manage softmax normalization pressure by creating designated receptacle tokens. That's an implementation detail of how self-attention handles probability mass conservation under the constraint that every query must produce weights summing to 1. Once you see it, you'll notice the same pattern in other transformer behaviors: the propensity to attend to separators, special tokens, and punctuation isn't random — these are learnable softmax valves. The model is routing excess attention probability to positions where it causes least harm.

StreamingLLM gives you a principled way to exploit this artifact. Keep the sinks, slide the window, don't evict the garbage collector. It's one of those fixes where the explanation is more valuable than the patch — because the explanation tells you what's actually happening inside the attention mechanism, not just which hyperparameters to set.