Tag
Distributed Systems
7 posts tagged Distributed Systems.
MegaScale: What ByteDance's 12,288-GPU Training Paper Actually Says
At 12,288 GPUs, you see roughly one hardware failure per day. Standard training frameworks have no answer for this. MegaScale is ByteDance's account of what actually breaks when you scale LLM training to 10,000+ GPUs — and the algorithm-system co-design changes that kept MFU above 55%.
Ring Attention: What the Near-Infinite Context Paper Actually Says
Extending context beyond what fits on one GPU isn't just a memory problem — it's a communication design problem. Ring Attention sequences the K/V data through a ring of devices and hides the transfers behind computation. Here's what that actually costs in production.
Dapper: What Google's Distributed Tracing Paper Actually Says
Every distributed tracing tool you use — Jaeger, Zipkin, OpenTelemetry — descends from one design decision Google made in 2010: sample at the trace root, not per-span. The paper explains why, and the failure modes it didn't fully solve.
Kafka: What the Original Paper Actually Says
The original Kafka paper from 2011 had no replication. A broker failure made all unconsumed messages permanently unavailable. The paper treats this as a limitation to fix later, not a deal-breaker. Understanding why explains more about Kafka's design philosophy than any architecture diagram.
MapReduce: What the Google Paper Actually Says
The 2004 Google paper that gave us Hadoop — and everything that replaced it — is worth reading not for the map/reduce abstraction itself, but for the fault tolerance model and the straggler insight. The failure modes are still the failure modes.
Pregel: What the Large-Scale Graph Processing Paper Actually Says
PageRank in MapReduce is O(iterations × full dataset reloads). Pregel fixes this by keeping the graph in memory across iterations and replacing disk I/O with message passing. The 'think like a vertex' model is the insight — BSP is the implementation.
Cassandra: What the Paper Actually Says
We had a Cassandra cluster where DELETE operations made reads progressively slower until queries timed out. Adding more disk space made it worse. The root cause is described precisely in the 2009 paper — but only if you understand that Cassandra cannot actually delete data.