GFS: what the Google File System paper actually says

Reading Ghemawat, Gobioff, and Leung (SOSP 2003) while investigating why our data pipeline had a small-files problem that no amount of tuning could fix.

The symptom was a Hadoop job that took 45 minutes to process a dataset that should have taken 5. The dataset had 12 million files, each around 4–8 KB. The NameNode was healthy, the cluster was otherwise idle, and there was no data skew. The job was spending nearly all its time in task initialization, not actual processing.

The root cause came down to a design decision in the Google File System paper from 2003, which HDFS closely approximates: chunk size is 128 MB by default, and every file — regardless of actual size — takes at least one chunk. 12 million 6KB files occupy 12 million 128 MB allocation slots in the NameNode's in-memory metadata store. The overhead wasn't I/O. It was metadata.

The fix was to compact the small files into larger Parquet files before the job ran. But understanding why this was the right fix, and why the alternative (tune the chunk size down) would have caused different problems, required going back to the GFS paper. The paper makes the small-files decision deliberately, on page three, in a section labeled "Assumptions."

GFS — "The Google File System", Ghemawat, Gobioff, and Leung, Google, SOSP 2003 — is not primarily a paper about building a distributed file system. It's a paper about choosing which file system properties to give up in order to get the ones that matter for a specific workload class. The contribution isn't the architecture. It's the argument that relaxing POSIX consistency and treating component failure as the normal case, not the exception, enables a system design that is both simpler and more reliable than the alternative.

The problem the paper is actually solving

By 2003, Google's workloads had specific characteristics that differed from what general-purpose file system designers assumed:

Files were large. Multi-gigabyte files were common. Multi-terabyte datasets composed of large files were the rule, not the exception.
Reads were mostly sequential. Web crawl data, log data, and intermediate MapReduce outputs are almost entirely sequential reads. Random access was rare.
Writes were almost entirely appends. Existing files were not modified; new data was appended to the end. Overwriting existing file content was uncommon.
Multiple clients appended to the same file concurrently. Log collection pipelines had many processes writing to a shared log. Coordinating these writes through a single writer was a bottleneck.
Hardware failed constantly. Google was running on commodity servers, not enterprise storage. Individual disk failures, machine failures, and network partitions were routine — happening daily across a cluster of thousands of machines.

The last two points are where GFS makes its most important choices. Commodity hardware failure is expected, so the system has to mask it transparently. Concurrent appends to shared files are the common write pattern, so the system has to make them efficient without requiring client-side coordination.

Traditional file systems made the opposite assumptions: files were small, access was random, a single writer modified a file at a time, and hardware was reliable. POSIX built an API on top of those assumptions. GFS does not try to be POSIX-compatible, and the paper is explicit about this.

The architecture: single master, many chunkservers

GFS has three components:

A single master stores all metadata: the file namespace, access control, mappings from files to chunks, and the current locations of chunk replicas. The master does not store chunk data — it stores only the metadata describing where data lives. All metadata is kept in memory. The master's disk holds a persistent operation log and periodic checkpoints, but the in-memory state is the authoritative view.

Chunkservers store the actual data. Files are divided into fixed-size chunks, each identified by a globally unique 64-bit chunk handle assigned by the master. Chunkservers store chunks as plain Linux files on local disk. By default, each chunk is replicated across three chunkservers.

Clients talk to the master for metadata operations (finding which chunkservers hold the chunks for a given file, getting a lease to write) and then talk directly to chunkservers for data operations. Clients cache chunk location metadata returned by the master. Data never flows through the master.

The single-master design looks like an obvious bottleneck from the outside. The paper's authors knew this. Their response was to minimize master involvement in every data path operation. Clients never read or write data through the master. They contact the master once, cache the response, and then talk directly to chunkservers until the cache expires. For large sequential reads — which are the dominant workload — a single master contact can amortize the metadata overhead over hundreds of megabytes of data transfer.

In practice, the master handled roughly 200–500 operations per second at the scale described in the paper. The metadata for the entire namespace fit comfortably in RAM (file/chunk names were a few bytes each, and location mappings were compact). The memory constraint was the actual scaling limit: at some file namespace size, the in-memory metadata exceeded what a single machine could hold. This is what eventually caused Google to replace GFS with Colossus, which distributes the master. The paper doesn't discuss this — it hadn't happened yet — but it's the first design flaw to materialize in production.

Why 64 MB chunks

The paper specifies 64 MB chunks, which was large compared to typical file system block sizes of 4–8 KB. The rationale has three parts.

Fewer master interactions per operation. A sequential read of a large file hits at most a few chunk boundaries. Each boundary requires a master lookup. With 64 MB chunks, a 640 MB sequential read requires ten chunk lookups. With 64 KB blocks, it requires ten thousand. At scale, the difference in master load is decisive.

Persistent TCP connections to chunkservers. Clients connect to chunkservers over TCP to read chunk data. With large chunks, a single connection serves many megabytes. The connection overhead is amortized. With small chunks, you'd be opening and closing connections constantly.

Smaller master metadata. 64 MB chunks mean fewer chunks per file, which means less metadata per file in the master's memory. A 1 TB file has 16,384 chunks. At 4 KB block sizes, it would have 256 million blocks. The master couldn't hold that in RAM.

The downside, which the paper acknowledges, is internal fragmentation for small files. A file smaller than 64 MB occupies a partial chunk. If many clients access that file, all accesses go to the same small set of chunkservers — creating hotspots. The paper's solution was to store small files with a lower replication factor and to route clients intelligently. In practice, GFS was not designed for small-file workloads, and using it for them creates exactly the metadata overhead problem I described at the start.

The consistency model: where the paper gets technical

The consistency model is the section of the paper that most engineers don't read carefully enough, and it's where most of the application-level complexity comes from.

GFS defines two properties for file regions after a mutation:

Consistent: all clients see the same data, regardless of which replica they read from.
Defined: consistent, and the mutation's full effect is visible (clients can see what was written).

A region can be consistent-but-undefined: all clients agree on the bytes, but the bytes don't correspond to what any single client wrote. This happens during concurrent writes from multiple clients.

The paper specifies the behavior for each mutation type:

| Mutation type | No concurrent writes | Concurrent writes | |---|---|---| | Write (to offset) | Defined | Consistent, undefined | | Record append | Defined (with at-least-once caveat) | Defined (with at-least-once caveat) |

Writes to specific offsets with concurrent writers produce garbage: the bytes from different writers are interleaved, but all replicas agree on the interleaving. This is consistent-but-undefined. Applications that write to specific offsets concurrently have to handle this themselves.

Record appends are the interesting case. GFS guarantees that the append will complete at least once, atomically, at an offset GFS chooses. If a record append fails on a subset of replicas, GFS retries it — meaning the record may appear more than once, at different offsets, across replicas. The paper calls this at-least-once semantics with possible padding and duplicate records.

The implication for applications: records must be self-describing and checksummed. The typical GFS-era pattern is to include a length prefix and CRC at the start of each record. A reader that encounters a duplicate or corrupted record detects it via the CRC and skips it. The application must be written to tolerate duplicates.

This is a significant shift from what POSIX-trained engineers expect. POSIX write semantics are: if write() returns success, the bytes are durably at the specified offset. GFS record append semantics are: if GCSappend() returns success, the bytes appear at least once somewhere in the file, and you'll need to scan and deduplicate when you read them back.

Data flow vs. control flow

The paper makes a distinction between control flow and data flow that is easy to overlook and important for understanding performance.

Control flow goes through the master. To write data, a client asks the master which chunkservers hold the replicas for a given chunk and which is the primary (holds the lease). The master responds with replica locations and a lease expiration. The client caches this and uses it for subsequent writes to the same chunk without contacting the master again.

Data flow is pipelined along a chain of chunkservers chosen for network proximity, bypassing the master entirely. When a client wants to write data to three replicas, it pushes the data to the nearest chunkserver. That chunkserver forwards it to the next nearest, and so on down the chain. Data moves along the chain concurrently — the first chunkserver starts forwarding data to the second while still receiving data from the client. This pipelining exploits full-duplex network connections and reduces the time to write data to all replicas.

The separation means the master's bandwidth is not involved in data movement. The master only transfers small control messages. All bulk data transfer happens directly between clients and chunkservers, and between chunkservers in the replication chain.

Leases and mutation ordering

When multiple clients write to the same chunk concurrently, GFS needs to serialize the writes so all replicas end up with the same data. The mechanism is a lease: the master grants a lease to one chunkserver for a given chunk, designating it the primary for that chunk. Leases last 60 seconds by default and are extended as long as mutation activity continues.

When a client wants to mutate a chunk, it gets the primary's identity from the master, sends data to all replicas (simultaneously, through the pipeline), and then sends a write request to the primary. The primary decides the serial order of concurrent mutations, applies them to its local replica, and forwards the ordered write requests to the secondaries. All replicas apply mutations in the primary's order. If a secondary fails to acknowledge, the primary reports an error to the client, which retries the failed region.

This means that chunk mutations are serialized through a single primary, eliminating the interleaving problem for sequential writes to a chunk. The inconsistency arises when clients write to different offsets — their writes don't go through a shared primary, so there's no ordering between them.

Fault tolerance

The paper's fault-tolerance section is where most of the engineering complexity lives.

Chunk replication: Three replicas by default, placed across racks. The master monitors replication levels via heartbeats and re-replicates chunks that drop below the target due to chunkserver failure, disk failure, or checksum errors.

Master recovery: The master writes every metadata change to an operation log on disk and on remote machines before acknowledging the change. The in-memory state can be reconstructed from the operation log. To bound recovery time, the master periodically checkpoints its state to a compact on-disk format. Recovery means loading the latest checkpoint and replaying only the operations that follow it. The paper targets recovery in under a minute for the cluster sizes described.

Shadow masters: During master recovery, GFS provides read-only access via shadow master replicas that trail the primary master by at most a fraction of a second. Shadow masters replay the operation log from the primary and maintain a slightly stale view of the namespace. Read-only operations (directory listings, file opens for reading) can proceed through shadow masters while the primary is down or recovering. Write operations require the primary.

Chunk integrity: Each chunkserver stores checksums alongside chunk data, at 64 KB granularity. Reads verify checksums; corrupted blocks are detected before they're served to clients. On a checksum mismatch, the chunkserver reports the error to the master, which schedules re-replication from a good replica and tells the client to read from a different chunkserver.

Heartbeats: The master monitors chunkservers via periodic heartbeat messages. A chunkserver that stops responding for too long is marked unavailable, and its chunks are scheduled for re-replication elsewhere. This handles machine failures, network partitions, and overloaded chunkservers without operator intervention.

The garbage collection mechanism is worth noting: when a file is deleted, GFS doesn't immediately free the chunk storage. Instead, the master logs the deletion and renames the file to a hidden name with a deletion timestamp. A background scan removes files that have been in the hidden namespace for more than three days, freeing the associated chunks. This lazy deletion approach simplifies crash recovery — if the master fails during a deletion, it doesn't matter; the deletion completes on the next scan. And it provides a three-day window to undelete files if the deletion was accidental.

Production tradeoffs no one mentions in the architecture overview

The single master is the first thing to fail at scale. The paper's cluster measurements show a metadata footprint of about 64 bytes per chunk. At 64 MB chunks, a petabyte of data requires roughly one billion chunks — 64 GB of master RAM, before indexing overhead. Google's production clusters exceeded this within a few years of the paper, which is why Colossus (GFS2) replaced GFS with a distributed master. If you're running HDFS and wonder why "NameNode capacity" becomes an operational concern, this is why.

Leases create write stalls. When a primary chunkserver fails mid-mutation, the master has to wait for the lease to expire (60 seconds by default) before granting a new lease to a different chunkserver. During this window, writes to affected chunks are blocked. The paper describes this as acceptable for their workload because most operations are reads and appends, not random writes. For write-heavy workloads, 60-second stalls on primary failure are not acceptable. HDFS inherits this behavior.

The relaxed consistency model requires application-level work. Every application built on GFS/HDFS that does concurrent writes has to implement its own deduplication logic. The typical pattern — length-prefixed, checksummed records — works, but it shifts the burden to every application developer. Teams that don't read the consistency section of the paper carefully write code that produces corrupt output silently: the writes succeed, the replicas are consistent, but the data interleaving makes the records unreadable.

Replication is synchronous for writes but the window for stale reads exists. When the primary forwards a write to secondaries, it waits for acknowledgment before returning success to the client. But reads can go to any replica without going through the primary, and a replica might be momentarily behind if it was slow to apply a mutation. The paper acknowledges this and calls it acceptable for their workload. For applications that write and then immediately read-back-to-verify, this can surface as surprising read-after-write inconsistency.

Tail latency at scale. GFS serves reads by sending requests to one replica (chosen for proximity). If that replica is slow — due to disk contention, network congestion, or GC — the read is slow. There's no hedging (sending the request to multiple replicas and taking the first response) in the original GFS design. For latency-sensitive applications, HDFS's later additions (speculative execution, rack-aware replica selection) address this, but the original paper doesn't.

Failure modes in practice

The most common operational failure mode I've seen is metadata explosion from small files.

HDFS NameNode memory is proportional to the number of files and blocks, not the total data volume. A petabyte of large Parquet files is manageable. A petabyte of small JSON files — each file one or two blocks — can exhaust NameNode heap even though it represents less total data. The fix is compaction: merge small files into larger ones before storing them. But this requires an explicit ETL step that many pipelines don't initially include, and the problem surfaces gradually as file counts grow, not as a hard failure at a known threshold.

The second failure mode is concurrent writer corruption from applications that don't implement the record-level integrity pattern.

A log aggregation system with 200 writers all appending to a single GFS file sounds like a natural fit — the paper explicitly describes this pattern. But if the log records aren't checksummed and length-prefixed, the bytes from different writers interleave across chunks in ways that produce valid-looking but semantically meaningless records. The file reads successfully. The bytes are consistent across replicas. The records are garbage. This failure mode shows up as "the log parsing job started producing nonsense" after a traffic spike, and it's hard to attribute without knowing the consistency model.

The third failure mode is silent checksum corruption on chunkservers with failing disks.

GFS's checksum verification runs on reads, not on periodic scrubbing. A block can be corrupted on disk between writes; the corruption is detected the next time the block is read. For cold data — blocks that are written and then not read for months — corruption can accumulate silently. HDFS added periodic block scrubbing to catch this. The original GFS paper relies on reads to surface corruption, which works for frequently-accessed data and doesn't for archival data.

When not to use GFS/HDFS

Small files. This is the direct consequence of the chunk size choice. If your median file size is below a few hundred megabytes, GFS is a poor fit. The metadata overhead at the NameNode is proportional to file count, not file size, and the internal fragmentation is significant. Object storage systems (S3, GCS) handle small files more efficiently because they separate the metadata path entirely from the data path and don't have fixed-block allocation.

Random access patterns. GFS is optimized for sequential reads and appends. Random writes require contacting the master to determine which chunk owns the target offset, potentially breaking the sequential pipeline pattern, and modifying arbitrary bytes in place within a chunk. The consistency guarantees for random concurrent writes are weak. Use a proper database — or a distributed key-value store like Bigtable — for random access workloads.

POSIX-dependent applications. Applications that assume POSIX file semantics — atomic create-on-missing, exact-once write, consistent read-after-write — will behave incorrectly on GFS/HDFS. The consistency model is documented and intentionally weaker. Tools like ls, stat, rename, and concurrent file-locking primitives behave in non-POSIX ways, or not at all.

Low-latency workloads. GFS optimizes for throughput, not latency. The master round-trip adds latency to every new chunk access. Large chunks mean reads can't start until an entire chunk is positioned. The architecture is suited to batch processing, not interactive queries. If your workload has p99 latency requirements in the hundreds of milliseconds, GFS is the wrong storage layer.

When you need strong consistency guarantees. GFS's "consistent but undefined" outcome for concurrent writes means the application has to handle data interleaving. If your application cannot tolerate this — financial records, configuration data, anything where exact-once semantics matter — you need a different system. Spanner, Bigtable, or a transactional database provide stronger guarantees, at higher cost.

What the paper actually gives you

GFS is a demonstration that the right design for a large-scale system often requires explicitly discarding requirements that seem fundamental.

POSIX consistency felt fundamental to file system designers in 2003. GFS's authors looked at their actual workload — sequential reads, append writes, concurrent producers on shared files, commodity hardware — and concluded that POSIX consistency was not only unnecessary but actively harmful to the design they needed. The relaxed model is not a concession; it's a design choice that enables simpler fault tolerance, simpler replication, and higher throughput.

The single master felt like a scalability non-starter. In practice, keeping metadata in memory and keeping data off the master path meant the single master scaled further than anyone expected. The constraint (metadata must fit in RAM) turned out to be a useful forcing function: it kept the namespace lean, prevented accumulation of metadata cruft, and made the failure domain simple. When the namespace eventually outgrew a single machine, Colossus distributed the master — but by then the architecture had proven itself at scale for years.

The insight that carries forward: know your workload before you design your consistency model. GFS's consistency choices look strange if you're building a general-purpose file system. They look exactly right if you're building a system for sequential-append workloads on commodity hardware. The paper's framing of assumptions first, design second, is the methodology worth stealing.

For your specific situation: if you're running Hadoop or Spark on HDFS and experiencing operational pain, the GFS paper explains most of the underlying constraints. Small-files problems, NameNode memory pressure, write stalls, and the lack of POSIX semantics in Hive/Spark file operations all trace back to design decisions made explicitly in this paper. Reading the original paper is faster than debugging the symptoms.

The Google File System — Ghemawat, Gobioff, Leung. SOSP 2003.