LoRA: what the low-rank adaptation paper actually says
Reading Hu et al. (Microsoft Research, ICLR 2022) while trying to fine-tune a 7B model on a machine with one 24GB GPU.
The math was simple enough. Fine-tuning in BF16 needs 2 bytes per weight for the model plus 12 bytes per weight for Adam optimizer states (momentum in FP32, variance in FP32, and a master weight copy). A 7B model is about 14 GB in BF16. Training needs roughly 14 + (12/2 × 14) = 98 GB. Our machine had 24 GB. The ratio was wrong by 4×, and buying four more GPUs was not on the table.
The standard answer at the time was adapters or prefix tuning or prompt tuning — various schemes for keeping the base model frozen and training only a small set of added parameters. They worked, but they added inference latency (adapter layers sit between attention blocks and require extra computation during generation) or required long learned prefix sequences that consumed sequence budget and were finicky to optimize. You were trading one problem for two others.
LoRA — "LoRA: Low-Rank Adaptation of Large Language Models", Hu, Shen, Wallis, Allen-Zhu, Li, Wang, Wang, and Chen at Microsoft Research, ICLR 2022 — solves the fine-tuning memory problem without adding inference latency. The mechanism is a decomposition of the weight update matrix that turns out to be both mathematically elegant and practically important for how the community now thinks about task-specific adaptation of large models.
The problem the paper is actually solving
Before the algorithm, there's a hypothesis. The paper builds on a 2020 result by Aghajanyan et al. showing that pre-trained language models have a low "intrinsic dimensionality" — that the loss landscape of fine-tuning tasks, when measured in the original parameter space, can be navigated almost as effectively using updates in a much smaller subspace.
The intuition: a model pre-trained on a large corpus has already learned most of the representations it needs. Task-specific fine-tuning isn't teaching the model new fundamental capabilities — it's more like adjusting a small number of directions in the already-learned representation space. The change in weights needed to adapt GPT-3 to, say, summarization might span only a few hundred effective dimensions even though the weight matrices have millions of entries.
If this is true — if the matrix of weight changes ΔW during fine-tuning has low effective rank — then constraining ΔW to be explicitly low-rank should work almost as well as full fine-tuning. LoRA makes this constraint explicit.
The algorithm
For each weight matrix W₀ ∈ ℝ^(d×k) in the model that you want to adapt, LoRA adds a parallel low-rank branch:
output = W₀x + ΔWx = W₀x + BAx
where B ∈ ℝ^(d×r) and A ∈ ℝ^(r×k), with rank r << min(d, k). During training:
- W₀ is frozen — no gradients flow through it, no optimizer states allocated for it
- A is initialized with a random Gaussian — standard initialization
- B is initialized to zero — so ΔW = BA = 0 at the start of training, preserving the pre-trained behavior
The zero initialization of B is load-bearing. It means LoRA adapters start as identity perturbations: the model's behavior at step 0 of fine-tuning is identical to the base model. This matters for stability — you're not starting from a perturbed state.
The forward pass adds a scaling factor α/r:
h = W₀x + (α/r) · BAx
where α is a hyperparameter. The r in the denominator cancels part of the effect of increasing rank — if you sweep r, you don't have to retune the learning rate from scratch. The paper typically sets α equal to r (scaling factor = 1) or to a fixed value and tunes r independently. In practice, most implementations default to α = r = 8 or α = 32, r = 16.
What this costs in parameters
For GPT-3 (d_model = 12288), applying LoRA to query and value projection matrices with r = 4:
W_q: (12288 × 4) + (4 × 12288) = 98,304 parameters per layer
W_v: same, 98,304 parameters per layer
× 96 layers = ~18.9M parameters total
The original GPT-3 paper reports 175 billion parameters. Adam optimizer states in FP32 for the full model: ~1.4 TB. Adam states for 18.9M LoRA parameters: ~227 MB. That's the practical difference.
The paper also compares against full fine-tuning on GPT-3 using GLUE, WikiSQL, SAMSum, and the E2E NLG benchmark. LoRA with r = 4 adapting only W_q and W_v matches or slightly exceeds full fine-tuning quality across these benchmarks. The result held even though the trainable parameter count was reduced by roughly 10,000×.
Which matrices to adapt
The original paper applies adapters to weight matrices in the self-attention mechanism: primarily W_q and W_v, with ablations showing the effect of including W_k and W_o. The finding was that distributing the rank budget across all four attention matrices (using lower r for each) outperforms concentrating it in just W_q and W_v at the same total parameter count.
The paper did not apply adapters to MLP layers. This was a deliberate choice for the experiments — not a claim that MLP adaptation is wrong. Subsequent work (including the QLoRA paper) found that applying adapters to all linear layers — attention projections plus the MLP gate, up, and down projections — consistently improves downstream quality, especially when working with quantized base models where the frozen weights are already lossy.
The practical heuristic that emerged from post-paper usage: apply to all linear layers except embedding tables and the LM head. The embedding layer isn't worth adapting with LoRA for most fine-tuning tasks (it requires the model to learn substantially new token representations, which LoRA's rank constraint limits). The LM head (the final projection from hidden state to vocabulary logits) is task-sensitive but typically fine with the base model's weights for classification and generation tasks.
No inference overhead — the merge
The cleanest property of LoRA, and the one that distinguishes it from adapter-based approaches, is that inference cost is identical to the base model after merging.
At inference time, compute W = W₀ + BA (scaling included). This is a single matrix addition done once before deployment. The resulting weight matrix W has identical shape to W₀. From the model architecture's perspective, nothing has changed — the merged LoRA model is structurally identical to the base model.
For deployment:
- Merged model: same size as the original, same FLOP count per forward pass, same latency
- Separate adapters (unmerged): requires adding BA computation in the forward pass, but B and A are small — the FLOP overhead is (d×r + r×k) additions per layer, compared to (d×k) in the main path. At r = 8 and d = 4096, this is 0.4% additional compute. Not zero, but negligible.
The unmerged approach matters when you're switching adapters frequently. You can keep the base model in GPU memory and hot-swap LoRA weights without reloading anything — the adapter matrices are small enough to transfer quickly. This is the foundation for multi-adapter serving systems like S-LoRA.
What the rank controls
Rank is the single most important LoRA hyperparameter, and the paper's discussion of it is worth reading carefully.
Higher rank means more expressive adapters but more parameters and more optimizer memory. Lower rank means fewer parameters but potentially underfitting. The paper's finding: for most downstream tasks tested, r = 4 to 8 is sufficient. Increasing r beyond 64 provides diminishing returns and can hurt (overfit on small datasets, or interfere with the pre-trained representations).
The paper includes an analysis of the learned ΔW matrices using SVD, comparing the top singular directions of the LoRA-learned ΔW against those of fully fine-tuned ΔW. The finding is that the LoRA adapters, even at r = 4, learn updates that are well-aligned with the top singular directions of the full update — they're approximating the most important directions well and leaving the low-magnitude directions behind. This is the empirical validation that the low-rank hypothesis holds.
There's also an interesting observation about the relationship between the A and B matrices. A has high "feature diversity" — its singular vectors span different directions. B shows higher overlap with the pre-trained weight space — it's learning to project the adapter's representation back into the model's existing feature directions rather than creating new ones. The adapter isn't learning from scratch; it's adjusting existing representations.
Production failure modes
Rank too low for the task shift. Adapting GPT-3 for SQL generation with r = 4 works because SQL generation is well within the model's prior. Adapting a general-purpose model for a specialized technical domain (clinical coding, formal verification) with r = 4 often leaves a gap. If you see validation loss plateau quickly but training loss continues improving, your rank is the bottleneck. Double it.
Forgetting the scaling factor. The α/r scaling is subtle. If you set r = 16 and α = 8, your effective learning rate for the adapter is halved relative to r = 8, α = 8. Libraries like peft handle this, but if you're implementing LoRA manually or sweeping hyperparameters, misconfiguring α produces confusing results that look like learning rate problems.
Not applying to MLP layers. The original paper's experiments didn't include MLP adapters, so teams often skip them. On instruction-following tasks and tasks requiring logical reasoning, MLP adaptation matters more than in the original paper's benchmarks. The hidden states that flow through the MLP carry different information than the attention-computed representations; restricting adaptation to attention projections leaves that channel frozen.
Adapter merging in multi-task scenarios. LoRA adapters are linear: W = W₀ + B₁A₁ + B₂A₂ is valid math, but the two adapters were trained assuming only one is active. In practice, adding adapter weights from different fine-tuning tasks produces unpredictable interference. Don't do arithmetic on LoRA adapters unless you know what you're doing — the literature on model merging (TIES, DARE, SLERP) is where to look if you actually need this.
Memory savings don't help if the base model doesn't fit. LoRA reduces optimizer memory but the base model still lives in GPU memory at full precision (or quantized precision). For a 70B model in BF16, that's 140 GB regardless of LoRA rank. LoRA doesn't let you fine-tune a model that doesn't fit — that's what QLoRA adds (4-bit quantization of the base weights). If your model fits but training doesn't, LoRA helps. If the model itself doesn't fit, you need QLoRA or sharding.
When NOT to use LoRA
Continued pre-training on domain data. LoRA constrains weight updates to be low-rank. If you're doing domain adaptation by training on billions of domain tokens, the update needed is not low-rank — it's a broad shift in the model's statistical model of language. Low-rank adapters will underfit significantly. Use full fine-tuning or at least full-rank fine-tuning with gradient checkpointing and sharding.
Tasks requiring new vocabulary. If your application needs the model to handle domain-specific tokens that weren't in the pre-training vocabulary (molecular SMILES strings, specialized notation, new programming languages), adding them to the tokenizer and training their embeddings requires touching the embedding table. LoRA's typical setup skips the embedding layer. You'll either need to train embeddings separately or use full fine-tuning.
Very small base models. Below ~1B parameters, the "savings" from LoRA are less compelling — full fine-tuning of a 350M model fits easily on a single GPU, and the expressivity of full fine-tuning is worth having. LoRA's regime is models large enough that full fine-tuning optimizer states become the binding constraint.
High-throughput multi-task serving without specialized infrastructure. If you have 50 customers each needing their own fine-tuned model and you want to serve all of them from a single GPU, LoRA makes this feasible — but only if you've built or adopted something like S-LoRA that handles adapter batching intelligently. Naively loading and swapping LoRA adapters per request gives you most of the operational complexity without the throughput benefits. Don't do multi-adapter serving unless you've accounted for the batching problem.
When you need provable quality guarantees. LoRA's quality is comparable to full fine-tuning on the benchmarks the paper tested. For novel tasks, novel domains, or production systems where failure modes are safety-relevant, "comparable on benchmark X" is not a quality guarantee. Full fine-tuning gives you the maximum possible update capacity; LoRA introduces an approximation whose quality on your specific task requires empirical validation.
The takeaway
LoRA's core contribution is making the intrinsic dimensionality hypothesis operational. The hypothesis — that fine-tuning updates are low-rank — was known. LoRA turned it into a training recipe: freeze the base weights, add trainable B and A matrices, zero-initialize B so you start from the pre-trained behavior, merge at inference for zero overhead.
The reduction from 1.4 TB of Adam optimizer state to ~200 MB is what made large model fine-tuning accessible on academic hardware and commodity cloud instances. Everything that followed — QLoRA's 4-bit quantization, S-LoRA's multi-adapter batching, DoRA's weight decomposition, PEFT tooling in Hugging Face — builds directly on this foundation. If you're adapting any model above 7B parameters and you're not starting from LoRA, you need a reason.
The rank is the lever. Start at r = 8 for standard instruction tuning, r = 16 for tasks that require broader adaptation, r = 4 if you're parameter-budget constrained. Apply to all linear layers, not just Q and V. Set α equal to r. Validate against full fine-tuning on a held-out set before calling it done.
The paper: "LoRA: Low-Rank Adaptation of Large Language Models," Hu et al., Microsoft Research. arXiv:2106.09685, ICLR 2022.