DeepSeek-R1: what reinforcement learning with verifiable rewards actually does

Reading DeepSeek-AI (January 2025) after a code-review agent started confidently explaining the wrong fix for a null pointer exception and no amount of prompting made it reliably backtrack.

The agent had all the context. Good retrieval. Decent system prompt. It knew about the function, the call stack, the test that was failing. It gave an explanation, the explanation was wrong, and it committed to the wrong explanation until you explicitly told it "that's incorrect." Only then would it reconsider.

The problem wasn't retrieval or context length. It was that the model had no training signal for being wrong. SFT trains on correct examples. RLHF from human preferences rewards responses that look good to annotators — and annotators often can't tell when code logic is subtly wrong. The model learned to be fluent and confident, not to be correct and self-aware of its own errors.

Reasoning models — OpenAI's o1, then DeepSeek-R1 — exist to address this. But they work differently from what most explanations imply, and the tradeoffs are significant enough that using them wrong is expensive.

"DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning" — DeepSeek-AI. January 2025.

The gap in the post-training taxonomy

Before DeepSeek-R1, the standard post-training toolkit was:

SFT: train on demonstrations of correct behavior. Efficient, predictable, requires good demo data.
RLHF / PPO (InstructGPT): train a reward model from pairwise human preferences, then optimize the policy with PPO. Adds comparative signal but requires reward model training, is PPO-unstable at scale, and is only as good as annotators' ability to judge quality.
DPO (Rafailov et al. 2023): derive a closed-form equivalent of the RLHF objective directly from preference pairs, eliminating the separate reward model. Simpler, more stable, widely adopted.

All three have the same blind spot: they rely on human-labeled data to define what "good" looks like. For tasks like mathematical reasoning or code correctness, humans often can't efficiently generate or evaluate the right answer at the level of detail needed. You can get annotators to judge "is this a well-written response," but you can't cheaply get them to verify whether a 200-line proof is correct at each step.

The paper's core contribution is showing that for tasks with verifiable outcomes — where you can automatically check correctness — you don't need human labels at all. The verification signal is the training signal.

DeepSeek-R1-Zero: pure RL from a base model

The paper introduces R1-Zero as a deliberate ablation: take a base model (DeepSeek-V3-Base), apply RL with verifiable rewards only, no supervised fine-tuning, no human demonstrations of reasoning chains.

The reward functions used:

Accuracy reward: does the final answer match the ground truth? For math, exact match or near-equivalence. For code, pass/fail against test cases.
Format reward: does the response follow the required structure? The model is prompted to output reasoning inside <think>...</think> tags, then a final answer. The format reward is binary — correct structure or not.

No process reward (scoring individual reasoning steps). No length bonus. Just: is the answer right, and is it structured correctly.

What emerged from this training without any explicit supervision on how to reason:

Extended chain-of-thought: the model spontaneously began generating longer internal monologues for harder problems, allocating more computation at inference time where it was needed.
Self-verification: the model started cross-checking its own work within the thinking trace, producing outputs like "let me verify this by substituting back into the original equation."
Backtracking: on problems where an initial approach failed, the model would abandon it mid-reasoning and try a different approach — without being told to, without seeing examples of this behavior.

The paper calls this the "aha moment" in training: a phase transition where the model's reasoning traces become markedly more sophisticated, corresponding to a jump in benchmark performance. This is not a gradual improvement. There's a point in training where the model figures out that spending more tokens on reasoning before committing to an answer is a better strategy for getting the reward.

The key implication: reasoning behavior is learnable from an outcome signal, not a process signal. You don't need to show the model how to think. You need to reward it when it's right.

GRPO: why they didn't use PPO

Standard PPO requires four models running simultaneously during training:

The policy (what you're training)
The reference model (the frozen SFT checkpoint, for the KL penalty)
The reward model (a separate fine-tuned LM)
The value model / critic (another LM-scale model estimating expected return)

At the scale of DeepSeek-V3-Base, this is approximately four copies of a 671B parameter model in memory. That's not practical.

R1 uses GRPO — Group Relative Policy Optimization (Shao et al. 2024) — which eliminates the critic entirely.

The GRPO approach: for each training prompt x, sample G outputs from the current policy {y₁, y₂, ..., yG}. Compute the reward for each: {r₁, r₂, ..., rG}. Then compute the advantage for each output as the normalized deviation from the group mean:

A_i = (r_i - mean(r)) / std(r)

This replaces the learned value function with a sample-based estimate. The model learns by seeing which of its own outputs, in direct comparison to each other, got better rewards. The gradient pushes the policy toward outputs that outperformed the group average and away from outputs that underperformed.

The practical effect: GRPO requires only two models (policy + reference), not four. Memory is roughly halved compared to PPO at the same model scale. The tradeoff is that GRPO needs larger batch sizes to get a good estimate of the group statistics — the group size G is a hyperparameter that trades memory for gradient signal quality.

The KL divergence constraint against the reference policy is handled via a clipped surrogate objective (same idea as PPO-clip) plus an explicit KL penalty term in the loss. This keeps the policy from drifting too far from the base model during RL, which matters because the base model's broad language capabilities are what make extended reasoning legible.

DeepSeek-R1: fixing R1-Zero's failure modes

R1-Zero works, but has documented problems in the paper:

Language mixing: when trained on multilingual data, the model sometimes mid-reasoning switches languages, producing thinking traces that mix English and Chinese.
Readability: some reasoning traces are repetitive, circular, or grammatically degraded — useful as a search process but not legible to humans or downstream consumers.
Cold start instability: starting RL from a pure base model means the early RL trajectories are noisy, which slows convergence.

The full R1 model uses a four-stage training process:

Cold start SFT: a few thousand examples of high-quality long chain-of-thought reasoning (human-curated or generated from a previous R1 checkpoint) are used to SFT the base model before any RL. This gives the RL training a better starting distribution — the model already knows approximately what reasoning traces look like, so RL can optimize them rather than discovering the format from scratch.
Reasoning-focused RL: GRPO on verifiable rewards as described above, but starting from the cold-start checkpoint rather than a raw base model. Convergence is faster and more stable.
Rejection sampling + SFT on mixed data: sample outputs from the RL-trained model, keep only the correct ones, and combine with high-quality data from non-reasoning domains (writing, factual QA, coding without formal test cases). Fine-tune on this combined set. This stage is about preventing capability regression on tasks that RL didn't optimize for.
Final RL on all domains: another round of RL on both reasoning (verifiable) and non-reasoning (preference-based) tasks using a combined reward signal. This aligns the model's behavior on creative and conversational tasks without sacrificing the reasoning gains.

The result is a model that reasons well on verifiable tasks and behaves well on everything else — without either capability cannibalizing the other.

Distillation: the practical shortcut most teams should use

Training a 671B model with GRPO is not something most teams do. The more immediately useful result from the paper is the distillation experiment.

The authors generated a large dataset of reasoning traces from the trained R1 model (600K examples, covering math, code, logic, and mixed domains). They then SFT-fine-tuned smaller open-source models — Qwen-7B, Qwen-14B, Qwen-32B, Llama-8B, Llama-70B — on this data.

The distilled 7B model beats GPT-4o on several math benchmarks. The distilled 70B model is competitive with o1 on AIME and Codeforces. These are base model SFT fine-tunes trained on R1's reasoning outputs, not RL-trained models.

This matters operationally: you can get most of the reasoning benefit from a much smaller model via distillation, without running RL at all. If you have access to R1 outputs (or any strong reasoning model's outputs), the distillation path is a standard fine-tuning pipeline on a 7B–70B model that fits on consumer hardware.

The paper also notes a negative result worth flagging: distilling from R1 into very small models (<7B parameters) with RL applied on top of distillation performs worse than distillation alone at those scales. The RL signal is too noisy for small models to use well. Distillation without RL is the right approach at small scale.

What this means for how you architect reasoning tasks

A few things that change once you have reasoning models available:

Test-time compute is a real dial. R1 genuinely produces better answers when allowed longer reasoning traces. The paper measures this directly: on AIME 2024, pass@1 improves from ~60% at short thinking budgets to ~80%+ at longer ones. If you have a hard reasoning task and latency is flexible, budget more tokens. If latency is constrained, accept lower accuracy. This is now a runtime parameter, not a model property.

Verifiable domains are different from non-verifiable domains. R1's training works because there's an automatic oracle. For code, test cases are the oracle. For math, symbolic verification or exact-match checking is the oracle. For tasks where "correct" is ambiguous — writing quality, factual nuance, strategic reasoning — R1's training didn't optimize for those, and the behavior is less predictable. The reasoning mechanism is present, but the RL reward signal wasn't available during training for those domains.

Self-correction is real but not free. Models trained with R1-style RL do genuinely revise their reasoning mid-trace. But this comes from the training objective, not from an architectural change. A standard model prompted to "think step by step" will also generate more tokens, but it's not doing the same thing — it's completing a reasoning-shaped sequence, not searching for a correct answer under an accuracy reward. The difference shows up on hard problems where a standard model will complete a plausible-but-wrong reasoning chain, while an R1-trained model is more likely to catch the error internally.

When not to use reasoning models

Latency-sensitive paths. R1 generates 10–30× more tokens than a standard model on hard problems. For an LLM serving stack, that means 10–30× the inference time and cost on those queries. If you have a P99 latency budget under 2 seconds, most reasoning model use cases are out.

Non-verifiable tasks. Using R1 for creative writing, summarization, or open-ended analysis doesn't buy you much — the model was RL-trained on verifiable outcomes. It will generate longer responses, but that's not the same as better reasoning. You're paying the latency cost without getting the accuracy benefit.

Straightforward tasks. Reasoning models are optimized for hard problems that benefit from extended deliberation. For simple extraction, classification, or templated generation tasks, a smaller standard model will be faster, cheaper, and roughly equivalent in accuracy. R1-distilled 7B models have the reasoning capability turned on globally — they'll reason about questions that don't need it.

When you need streaming output. Reasoning traces are interleaved with the final answer inside <think> tags. If you're streaming to a user interface, you either stream the thinking trace (which exposes model internals and looks odd) or buffer until the answer section starts. Buffering kills the streaming UX. Some serving frameworks strip think tags before streaming, but this adds latency in a different way.

When the verification oracle doesn't exist. The whole RLVR framework depends on being able to automatically check answers. If you're fine-tuning your own reasoning model and your domain doesn't have reliable test-case execution or formal verification, you're back to RLHF with preference labels, and R1's specific training approach doesn't apply.

The actual shift

What the paper demonstrates isn't just a new model. It's that reasoning behavior — planning, backtracking, self-verification — can emerge from a reward signal on outcomes, without any human labeling of the reasoning process itself.

The practical consequence is a new point on the cost-accuracy curve that didn't exist before: you can now get significantly better accuracy on hard reasoning tasks by generating more tokens at inference time. This is a different kind of capability than making a model larger or training it on more data. It's trading latency and token cost for correctness, at inference time, on a per-query basis.

That's a useful thing to have. It's also a capability that's easy to misapply by routing queries to it that don't benefit from extended reasoning, and expensive to misapply by running it on latency-sensitive paths. The architecture question isn't "should we use a reasoning model" but "which queries in this system actually need one, and what's the latency budget."

Further reading: RLHF: What the InstructGPT paper actually says for the foundation this builds on. DPO: What the paper actually says for the alternative post-training path that RLVR complements rather than replaces.