Topic
Fine-Tuning & Adaptation
Adapting pre-trained models to your task without burning a cluster — LoRA and its variants, quantized training, and the tradeoffs that decide quality.
The Llama 3 Herd of Models: What the Paper Actually Says
Llama 3's 405B benchmark numbers are fine. The paper is actually about something more useful: what decisions you make when you can train on 15T tokens across 16K H100s, and which of those decisions transfer to your deployment.
Switch Transformers: What the Sparse MoE Scaling Paper Actually Says
Every modern large model — Mixtral, DeepSeek, Gemini — routes tokens through sparse experts. The design decisions in all of them trace back to one 2021 Google Brain paper. The paper is worth reading because the failure modes live in the routing logic, not the math.
T5: What the Text-to-Text Paper Actually Says
Every instruction-tuned model today owes something to T5's core idea: every NLP task is just sequence-to-sequence. But the paper's real contribution is a systematic ablation of what actually helps in transfer learning — and several of the answers are counterintuitive.
DeepSeek-V3: What the Frontier-on-a-Budget Paper Actually Says
DeepSeek-V3 trained a 671B-parameter frontier model for ~$5.5M. The paper is less about model quality and more about whether the training stack itself is the bottleneck — and how to engineer around it.
Toolformer: What the Paper Actually Says
A 6.7B model beats GPT-3 175B on math by learning to use a calculator. Toolformer's self-supervised training pipeline is the interesting part — and it's more constrained than the demos imply.
Ring Attention: What the Near-Infinite Context Paper Actually Says
Extending context beyond what fits on one GPU isn't just a memory problem — it's a communication design problem. Ring Attention sequences the K/V data through a ring of devices and hides the transfers behind computation. Here's what that actually costs in production.
MegaScale: What ByteDance's 12,288-GPU Training Paper Actually Says
At 12,288 GPUs, you see roughly one hardware failure per day. Standard training frameworks have no answer for this. MegaScale is ByteDance's account of what actually breaks when you scale LLM training to 10,000+ GPUs — and the algorithm-system co-design changes that kept MFU above 55%.
LoRA: What the Low-Rank Adaptation Paper Actually Says
Full fine-tuning GPT-3 requires roughly 1.4 TB of optimizer state. LoRA gets trainable parameters down to ~4.7M with comparable quality — by exploiting a property of pre-trained models that most engineers know about but don't fully reason from.