Mistral 7B: what the sliding window attention paper actually says

Reading Jiang et al. (Mistral AI, 2023) after watching KV cache blow out memory on a long-context chat application.

The failure was predictable in retrospect. We had a customer support application where users would have extended back-and-forth conversations — 30-40 turns, technical troubleshooting, often 10,000 tokens by the time the issue resolved. We were serving a 7B model on A10G GPUs (24 GB VRAM). Model weights in BF16: ~14 GB. That leaves 10 GB for KV cache and activations.

KV cache for a 10,000-token session on a standard 7B model with 32 layers, 32 KV heads (no GQA), 128-dim heads, in FP16:

10,000 tokens × 32 layers × 2 (K + V) × 32 heads × 128 dims × 2 bytes
= 5.24 GB per session

That's one session. Two concurrent long sessions fills the remaining VRAM. The model started throwing OOM errors whenever two users hit the 8K-token mark simultaneously.

The naive solutions are all bad. Truncate context: you lose earlier turns that are often diagnostically important. Cap the context window: same problem, you've just moved when it breaks. Quantize KV cache to INT8: roughly 2× savings, plus quality regressions on long-context recall. Upgrade to H100s: you've doubled infrastructure cost to accommodate a math problem you haven't actually solved.

Mistral 7B — "Mistral 7B", Jiang et al., Mistral AI, 2023 — addresses this with two mechanisms that together fundamentally change the KV cache arithmetic: sliding window attention (SWA) and a rolling buffer cache. Combined with grouped-query attention (GQA), the result is a model that uses a bounded, constant amount of KV cache memory regardless of how long the sequence grows, while outperforming Llama 2 13B on every standard benchmark the paper measures.

The bounded memory claim is worth pausing on. It's not "lower memory for the same context length." It's that KV cache size is constant as sequence length grows beyond the window. That's a qualitative change in the scaling behavior.

What sliding window attention actually does

Standard attention lets every token attend to every previous token. At sequence length N, this means N KV pairs per layer, per session — a cache that grows without bound.

Sliding window attention restricts each token to attend to at most W previous tokens. Mistral uses W = 4096. Token at position i attends to positions max(0, i−W) through i. Outside that window: no direct attention, no stored KV pairs.

The immediate concern is obvious: what happens to information earlier in the context? If token 8000 can't directly attend to token 500, does the model lose it?

The multi-layer structure is the answer. At layer 0, position i sees [i−W, i]. At layer 1, position i attends to [i−W, i] — but each of those positions already incorporated information from [i−2W, i] at the previous layer. At layer 2, the effective reach is [i−3W, i]. Information propagates through accumulated hidden states, not direct attention, but it does propagate.

With L layers and window size W, the theoretical receptive field at the top layer is:

receptive_field = L × W = 32 × 4096 = 131,072 tokens

Mistral 7B's context window is 8,192 tokens — well within this range. Every position in the context is reachable from every other position. The question is whether multi-hop propagation quality matches direct attention, and the paper's benchmarks say: for most standard tasks, yes. What it says about harder tasks is in the failure modes section below.

The rolling buffer cache

Sliding window attention bounds attention computation to W tokens per position. But without additional machinery, the KV cache would still grow as the sequence grows — you'd just stop using old entries while still allocating memory for them.

Mistral solves this with a circular buffer. The cache has exactly W slots, and the KV values for position i are stored at slot i mod W:

cache_slot(i) = i mod W

When position 4097 is computed, it overwrites the slot previously used by position 1 (4097 mod 4096 = 1). The cache has exactly W = 4096 slots permanently, regardless of total sequence length.

The memory cost per session becomes fixed:

W tokens × L layers × 2 (K + V) × n_kv_heads × head_dim × bytes
= 4096 × 32 × 2 × 8 × 128 × 2 bytes   (Mistral uses 8 KV heads via GQA)
≈ 0.5 GB per session

Back to our failing deployment: 10 concurrent sessions at any context length → ~5 GB total KV cache. The same sessions on the original model at 10K tokens → 52.4 GB. The problem is gone, not managed.

The GQA contribution compounds the savings. Mistral uses 8 KV heads instead of 32 query heads — a 4× reduction in KV cache size over standard multi-head attention, stacked on top of the SWA-imposed bound. The 0.5 GB per session number includes both factors.

Chunked prefill for long inputs

The rolling buffer handles KV cache during generation. Prefill is a separate problem: you need to process the input prompt, and if the prompt is longer than W tokens, you need a strategy for positions that are outside one window of each other.

Mistral uses chunked prefill: split the prompt into chunks of size W, process each chunk sequentially, and carry forward the hidden state. Within each chunk, SWA applies normally — tokens attend to up to W previous positions, including positions from the prior chunk via the rolling buffer.

For an 8K prompt, this means two 4K chunks rather than one 8K block. Computational cost is comparable, but peak attention allocation is W × W = 4K × 4K per chunk rather than 8K × 8K for the full prompt. You never allocate the quadratic matrix for the full sequence.

What the benchmarks actually show

The paper compares Mistral 7B against Llama 1 (7B, 13B, 34B) and Llama 2 (7B, 13B) across HellaSwag, PIQA, WinoGrande, ARC, MMLU, MBPP, HumanEval, and GSM8K.

Results worth internalizing:

• Mistral 7B vs. Llama 2 7B: Mistral wins on every benchmark. Margins range from modest (HellaSwag: 81.0 vs. 77.8) to large (GSM8K math reasoning: 52.2 vs. 24.6). A 2× improvement in math reasoning at the same parameter count is not a rounding error.

• Mistral 7B vs. Llama 2 13B: Mistral 7B wins on all benchmarks. A 7B model outperforming a 13B model of the same generation, across a broad set of tasks.

• Mistral 7B vs. Llama 1 34B: Mistral 7B wins on most benchmarks. A 7B model matching or beating a 34B from the prior generation.

The paper attributes most of the quality gains to better training data and procedure, not the architectural changes. SWA, GQA, and the rolling cache are efficiency improvements — they don't explain why Mistral 7B is so much stronger than Llama 2 7B on reasoning benchmarks. What the architecture does is make the 7B parameter point economically viable for the quality it achieves: you can serve this model on a single A10G without KV cache exploding your memory budget.

Production tradeoffs and failure modes

SWA degrades on strict long-range retrieval. The receptive field calculation (L × W = 131K) is theoretical — actual information quality through multi-hop propagation is lower than direct attention. On needle-in-a-haystack tasks where the model must recall a specific value from position 500 when generating at position 7000, and position 500 is outside the current window, the model relies entirely on multi-layer hidden state propagation. In practice, Mistral 7B shows measurable degradation on these tasks compared to full-attention models at the same scale. If your task requires reliably attending to specific information anywhere in a long context, this architecture has a real cost.

The rolling cache evicts silently. When a session exceeds W tokens, old KV entries are overwritten. No error is raised — the model simply cannot directly attend to those positions. If a user states their account number at turn 1 and references it at turn 40 (well past 4096 tokens), the model has lost direct access to that value. It may still answer correctly if the information propagated through intermediate states, or it may not. The failure is probabilistic, not deterministic, which makes it especially hard to catch in testing: test conversations are usually shorter than W, so you never hit the eviction boundary.

Chunked prefill has a seam at chunk boundaries. For long prompts processed in chunks, attention within each chunk is accurate. Cross-chunk relationships are mediated by the hidden state at chunk boundaries, not direct attention. If two facts that need joint reasoning appear in different chunks and are far apart, the model has reduced ability to combine them compared to full-attention processing. This matters most for prompts that include multiple long reference documents that need to be compared or synthesized.

The window size is fixed at training time. W = 4096 is baked into the architecture and rolling buffer. You can't extend the window for requests that need it. If you need exact long-range attention for a subset of your workload, you need a different model — not just a larger cache allocation.

GQA's shared KV heads can affect generation diversity. With 8 KV heads shared across 32 query heads, four query heads share every key and value representation. At high concurrency with structurally different prompts, this architectural compression limits how differently those query heads can attend to the same position. For most generation tasks this is unnoticeable, but for tasks requiring fine-grained disambiguation — distinguishing between multiple similar entities in a long context, for example — the 4× reduction in KV expressivity can manifest as confusion the base architecture wouldn't have had.

When NOT to use Mistral 7B (or SWA architectures generally)

Don't use SWA when you need exact attention over full context length. Legal document analysis, contract comparison, long-form structured extraction — tasks where the model must accurately attend to any position in a potentially long document. SWA works well on average but has concrete degradation on explicit cross-reference tasks. For this use case, use a full-attention model with PagedAttention for memory management — you want the KV cache growth, controlled.

Don't use the rolling cache when conversation history must be reliably preserved. If users expect the model to accurately recall anything said at any point in a conversation, a rolling buffer that silently evicts old context will produce hard-to-diagnose failures. Either implement explicit context management — summarize old turns, inject key facts into the active window — or use an architecture that makes the context boundary explicit.

Don't choose Mistral for memory efficiency when the bottleneck isn't context length. If your sessions are short and you just want a model that fits in 24 GB, there are more direct paths: AWQ or GPTQ on a larger model may give better quality at similar or lower memory footprint. SWA's memory advantages are most valuable when long context is the bottleneck, not model size.

Don't use benchmark numbers as a proxy for your task. Mistral 7B outperforms Llama 2 13B on the paper's benchmarks. Those benchmarks are not your task. If your task involves long-context coherence, complex instruction following, or domain-specific knowledge that wasn't well-represented in Mistral's training data, measure on your own data before making deployment decisions.

Why this architecture matters beyond Mistral

The rolling buffer KV cache solves a problem that gets worse as context windows grow. Full-attention architectures with 128K or 1M context windows have KV cache costs that are incompatible with high-concurrency serving without heroic memory management — PagedAttention, offloading, aggressive quantization. The costs scale linearly with context length, which means every request that uses more context is a more expensive request to serve.

SWA with a rolling cache breaks that scaling relationship. Memory cost is bounded at W regardless of session length. You can accept long sessions without budgeting for their worst-case KV cache footprint — the worst case is the same as the common case.

Mistral's contribution is showing this works at a competitive quality level with straightforward architectural changes. The design space includes sparse attention patterns, linear attention approximations, and state-space models like Mamba — each with different tradeoffs in quality, training complexity, and inference flexibility. What Mistral demonstrates is the lower bound on complexity for solving the context-length memory problem: a standard transformer with a fixed window and a circular buffer gets you most of the way there.

The practical question for any production deployment is whether you're paying for KV cache growth you don't need. If your longest sessions are 4K tokens, SWA gives you nothing. If sessions routinely exceed the window and you need reliable recall of the full history, SWA actively hurts you and you need to build around it. But if your sessions are long, your memory is tight, and your task tolerates some degradation in very-long-range recall — Mistral 7B is a case study in how much you can get for how little, if you pick the right architectural trade.