ZeroFS vs. Amazon S3 Files

2026-07-11T00:00:00+00:00

Amazon S3 Files and ZeroFS expose POSIX filesystems backed by object storage, but the shared interface hides opposite bucket layouts. The choice turns on the role of the bucket: if files must remain ordinary S3 objects, S3 Files preserves that identity; if the bucket can be an internal persistence layer, ZeroFS trades direct S3 access for packing, compression, and client-side encryption.</p>

flowchart TB subgraph CLIENTS["Client Layer"] NFS["NFS Client"] P9["9P Client"] NBD["NBD Client"] WEB["Web Browser"] end subgraph CORE["ZeroFS Core"] NFSD["NFS Server"] P9D["9P Server"] NBDD["NBD Server"] WEBUI["Web UI"] VFS["Virtual Filesystem"] SEG["Segment store: compressed, encrypted file-data frames"] SLATE["LSM tree: metadata and 32-byte extent pointers"] CACHE["Local Cache"] NFSD --> VFS P9D --> VFS NBDD --> VFS WEBUI --> VFS VFS --> SEG VFS --> SLATE SEG --> CACHE SLATE --> CACHE end subgraph BACKEND["Storage Backend"] SEGOBJ["Immutable segment objects"] SSTS["Metadata SSTs and manifest"] S3["S3 Object Store"] CACHE --> SEGOBJ CACHE --> SSTS SEGOBJ --> S3 SSTS --> S3 end NFS --> NFSD P9 --> P9D NBD --> NBDD WEB --> WEBUI </pre></div>

ZeroFS keeps file data and filesystem metadata on separate paths until both reach the object store.</figcaption> </figure>

Both mounts use the client page cache. The write-path row below starts after the client sends the write to the server.</p>

</th> Amazon S3 Files</th> ZeroFS</th></tr></thead>

Storage model</td> AWS-managed high-performance storage synchronized with a bucket; internal layout not documented</td> An LSM tree and immutable data segments on object storage</td></tr>

Object layout</td> One file maps to one S3 object</td> Metadata in an LSM tree; file-data frames packed into segments</td></tr>

Write path after client cache</td> NFS write to high-performance storage, durable immediately; asynchronous S3 export after write inactivity</td> Over 9P, fsync</code> uploads data segments and flushes LSM metadata to object storage before returning success</td></tr>

Cold read and read-ahead</td> Linux NFS read-ahead, directory metadata import, optional small-file import, or direct S3 read</td> LSM lookup, then adaptive object and cross-segment frame prefetch</td></tr>

S3 API access to files</td> Yes; filesystem changes appear after asynchronous export</td> No; reading files requires ZeroFS and the encryption password</td></tr>

Client interface</td> NFS 4.1/4.2</td> 9P or NFS for files; NBD for blocks</td></tr>

Object-store choice</td> Amazon S3</td> Amazon S3, S3-compatible stores, Azure Blob, or Google Cloud Storage</td></tr>

Cost model</td>

S3 plus high-performance storage, file access, and synchronization charges</td>

Object storage and requests, plus the compute and cache running ZeroFS</td></tr> </tbody> </table> </div>

Object interoperability</h2>
Keeping that one-to-one mapping means a file written through the mount eventually becomes a normal S3 object</a>. Export begins after 60 seconds without a write, so continued writes postpone S3 visibility. Once export completes, existing tools can read the object with `GetObject</code>, and S3-side changes flow back into the filesystem. If both sides modify the same file before synchronization, S3 wins and the file-side version moves to lost+found</code>.</p>`
`A ZeroFS pathname cannot be fetched with the S3 API, and a segment cannot be scanned as Parquet. In exchange, small files need neither one data object nor one PUT each: their extents are compressed, encrypted, and packed together. The bucket and raw local cache contain ciphertext, so mounting or recovery requires ZeroFS and its encryption password. The` `encryption documentation</a> lists the structural metadata that remains visible.</p>`
`If other applications need immediate S3 visibility after a filesystem write, neither model provides it: S3 Files exports asynchronously, while ZeroFS never exposes mounted files through the S3 API.</p>`

Cold access</h2> The first S3 Files access can trigger an import: listing a directory loads every object's metadata and asynchronously copies files below the import threshold, 128 KiB by default, into high-performance storage. AWS says a first listing of 1,000 objects may take several seconds</a>. Larger files stay in S3, and reads of at least 1 MiB go directly to S3</a>.</p> ZeroFS instead populates a local RAM and disk cache</a> from reads and uploads. Newly sealed segments enter the cache from bytes already in hand, so read-after-write needs no GET; the cache is not write-back, and writes still reach object storage when segments are sealed and metadata is flushed. A cold miss resolves the requested extents through the LSM tree and coalesces adjacent frames into ranged GETs. It pays the first object-store round trip without importing the rest of the file or every small file in the directory; once the working set fits locally, data reads generate few S3 requests.</p> Both paths add read-ahead for sequential traffic: ZeroFS prefetches within and across segment objects, while S3 Files relies on Linux NFS read-ahead alongside its direct-S3 routing.</p> Cost</h2> S3 storage and request charges apply in either case. S3 Files also charges for its high-performance storage tier: resident storage per GB-month and reads and writes per GB. The AWS pricing example</a> current at publication uses these rates:</p> S3 Files charge</th> Rate used in AWS's example</th> When it applies</th></tr></thead> High-performance storage</td> $0.30 per GB-month</td> Files have a 10 KiB minimum billable size</td></tr> File reads</td> $0.03 per GB</td> Reads from high-performance storage, including exports to S3</td></tr> File writes</td> $0.06 per GB</td> Writes to high-performance storage, including imports from S3</td></tr> </tbody> </table> </div> The amount resident in high-performance storage depends on the configured import threshold and expiration window</a>. Listing imports metadata and eligible small files, reads can load more data, and file metadata does not expire</a>. Imports are metered as filesystem writes and exports as reads; the minimum operation sizes</a> are 32 KiB for file data and 4 KiB for metadata. AWS reports resident bytes and inodes in CloudWatch</a>, but total bucket size alone cannot forecast the bill.</p> Illustrative model: store 10,000 GiB and read it once</h3> This is an illustrative scenario, not a general cost comparison.</strong> It uses 10,000 AWS billable GB (GiB) of logical data in S3 Standard in us-east-1</code>, at $0.023 per GB-month</a>. The direct case assumes 1 MiB reads from data already in S3; the resident case assumes reads below 1 MiB with all data also in high-performance storage. Internet egress is excluded.</p> Case</th> Storage/month</th> Read once</th> Illustrative subtotal</th></tr></thead> ZeroFS (2:1)</td> ~$115 + overhead</td> S3 GETs</td> ~$115 + infrastructure</td></tr> ZeroFS (1:1)</td> ~$230 + overhead</td> S3 GETs</td> ~$230 + infrastructure</td></tr> S3 Files (1 MiB direct reads)</td> ~$230 + metadata</td> ~$1.17 + S3 GETs</td> ~$231 + metadata and requests</td></tr> S3 Files (small resident reads)</td> $3,230</td> $300</td> $3,530 +$600 if imported</td></tr> </tbody> </table> </div> The ZeroFS figures are estimates: compression reduces payload bytes, while metadata, temporary compaction data, and requests add cost. At an ideal 8 MiB GET size, reading the file data costs about $0.26 at 2:1 compression or $0.51 at 1:1, before metadata and short or random reads. ZeroFS also requires one server, or two for high availability, plus local disk cache; each node needs about 2 GB of RAM beyond its configured memory cache. S3 Files may therefore cost less for large-file streaming. The difference grows when data is imported into, or written through, high-performance storage: writing 10,000 GiB through S3 Files adds $600 in filesystem writes and $300 for export, while a month of residency adds $3,000.</p> Actual costs depend on file count, compression, I/O size, residency, fsync</code> frequency, and infrastructure. S3 Files also requires bucket versioning</a>, so noncurrent versions need an appropriate lifecycle rule.</p> fsync</code> and S3 visibility</h2> Durability and S3 visibility are separate events in S3 Files. Writes to high-performance storage are durable immediately</a>, but export starts only after approximately 60 seconds without a write</a>; calling fsync</code> does not make the object immediately visible through the S3 API. ZeroFS has no second visibility event because there is no export step. Over 9P, a successful fsync</code></a> returns after object storage acknowledges the data segment and its LSM metadata, leaving the file recoverable after a cold restart.</p> Rename</h2> S3 has no directories or atomic rename for general-purpose buckets</a>. S3 Files renames on the mount, then copies each affected object to a new key and deletes the old one; during a directory rename, both prefixes can be visible. AWS says synchronizing 100,000 renamed files takes a few minutes</a>. ZeroFS changes only its LSM directory entries, with no copy per descendant and no S3 prefix exposed to object clients.</p> The mount points may look similar, but the buckets make different promises: S3 Files preserves ordinary objects for the surrounding S3 ecosystem; ZeroFS treats object storage as the private substrate of the filesystem.</p> Making POSIX filesystems replicated and highly available 2026-06-28T00:00:00+00:00 ZeroFS stores a POSIX filesystem in a log-structured database in S3. On one node, acknowledged but unflushed writes live only in memory, and the filesystem is unavailable whenever that node is down. A standby must cover both gaps without weakening the durability promised by fsync</code>.</p> Where a single node runs out</h2> Object storage is durable, but each request is slow and billable. Committing every filesystem operation directly would add tens of milliseconds and a PUT charge to each write. ZeroFS instead buffers writes in an in-memory memtable and flushes batches on fsync</code>, when the memtable fills, and on a periodic timer. A write()</code> returns once the data reaches the memtable; fsync()</code> pushes it to S3.</p> This creates two failure modes. A crash before a flush loses acknowledged writes, which POSIX permits because fsync</code>, not write</code>, is the durability boundary. Data already flushed remains in S3, but no server exposes the filesystem while the node is down. A standby can retain the in-memory tail and take over serving, provided it never becomes a second writer.</p> Only one writer, ever</h2> Two nodes over one bucket can split-brain: both believe they are the leader and write concurrently. Once their updates interleave, the filesystem is corrupt. Preventing that cannot depend on timing, network reachability, or synchronized clocks.</p> The storage layer chooses the single writer. Opening a database for writing conditionally updates its manifest, increments a writer_epoch</code>, and fences the previous holder. The old leader's next manifest write fails and it stops. Any SST files it uploaded in the meantime remain unreferenced and are later reclaimed by garbage collection. During a race, both nodes may briefly claim leadership, but only one epoch can commit.</p> Fencing uses conditional writes, supported directly by S3, Azure Blob, and Google Cloud Storage. For S3-compatible stores without that primitive, ZeroFS uses Redis. At startup, each node also asks its peer which one is active instead of trusting its configured role.</p> </div> Why two nodes, not three</h2> Raft and Paxos clusters use an odd number of members because the nodes vote on leadership and committed state. ZeroFS has no node quorum. A conditional update to the manifest in the bucket decides which node owns the writer epoch.</p> The object store's linearizable compare-and-swap can choose one winner among any number of contenders. One ZeroFS node serves; the second holds the un-fsync</code>'d tail and can take over. A third would add no voting safety because the vote already happens in the bucket.</p> Keeping a standby in step</h2> Replication is semi-synchronous and ordered. For each write, the leader sends it to the standby, waits for an acknowledgement, applies it locally, and then answers the client. Anything visible to a client has therefore reached the standby. Unapplied writes remain in a tail buffer for replay during takeover.</p> If the standby falls behind or disconnects, the leader enters Solo instead of blocking the filesystem. Replication resumes when the standby returns. Solo writes have single-node durability: writes not yet flushed have no second copy. Fencing still prevents another writer from committing.</p> When the leader dies</h2> The leader sends a heartbeat every 100 ms with its writer epoch. After roughly two seconds without one, the standby opens the database for writing, fences the old leader, replays its buffered tail, and starts serving. Heartbeats from the old epoch are ignored. If a standby with an unreplayed tail hears from a stale leader, it promotes immediately rather than waiting for the timer.</p> sequenceDiagram participant C as Client participant L as Leader participant S as Standby participant O as Object storage Note over C,O: Steady state C->>L: write L->>S: ship (semi-sync) S-->>L: replicated L-->>C: ack (after local apply) L->>O: flush on fsync L->>S: heartbeats (writer epoch) Note over L: Leader crashes or is partitioned L--xS: heartbeats stop Note over S: no heartbeat for ~2s (takeover TTL) S->>O: open as writer, fence old leader S->>S: replay buffered tail Note over S: now serving as the new leader C->>L: in-flight write L--xC: fail, timeout, or not-leader C->>S: reroute, resend with op-id S-->>C: definitive result (deduplicated) Note over L: a returning old leader stays fenced </pre></div> A leader failure end to end: steady state, takeover, and the client's rerouted retry.</figcaption> </figure> Clients know both addresses and reroute when one node stops serving. A request in flight during failure may already have committed, so every mutating call carries a stable operation id. The client can retry a mkdir</code> or rename</code> on the new leader without duplicating the effect.</p> The old leader must stop reads as well as writes. A node serves only while its lease is valid, and the lease follows the storage engine's view of the database. A manifest poll sees the new epoch and closes the stale database within one interval, even if the old leader is idle.</p> When fsync can't prove durability</h2> Replication reduces but cannot eliminate the un-fsync</code>'d window. Losing both nodes, or losing a leader in Solo, discards acknowledged writes that never reached S3. A successful fsync</code> means every write acknowledged on that descriptor is in object storage and, under HA, survives failover. If recovery did not carry those writes, fsync</code> returns ESTALE</code>.</p> A lineage token</strong> identifies an uninterrupted durable history. A connected standby that replays the leader's tail retains the token. A cold restart or takeover after Solo gets a new token because it cannot prove that it inherited every write. Before acknowledging its first Solo write, the leader marks the lineage un-inheritable in object storage. The client tags un-fsync</code>'d changes with the current token; on fsync</code>, the leader flushes and compares the oldest outstanding token with the live one. A mismatch fails the call.</p> What happened</th> Token</th> fsync result</th></tr></thead> Connected leader fails</td> Kept: the standby held the writes and replayed the tail</td> Succeeds after failover</td></tr> Both nodes lost before a flush</td> Regenerated: a cold restart can't prove it inherited anything</td> ESTALE</code>: the writes were only in memory</td></tr> Takeover after Solo</td> Regenerated: the Solo leader tainted the lineage first</td> ESTALE</code>: the new leader never got the writes</td></tr> </tbody> </table> </div> The token check is per descriptor but covers the whole file across open handles, plus metadata, directory entries, and both sides of a rename</code>. On a mismatch, the next fsync</code> returns ESTALE</code>: work since the last successful fsync</code> is gone. Client libraries expose this as a distinct stale-handle error.</p> Trying to break it</h2> ZeroFS tests these guarantees with Jepsen</a>. The single-node suite generates filesystem operations over a 9P mount and checks them against a reference model. Its crash mode kills the server, drops the memtable, recovers from object storage, and verifies the result against the last fsync</code>. This suite runs in CI on every change.</p> The HA suite runs a leader and standby over MinIO while a nemesis kills either node or both, partitions the pair, pauses the object store, and freezes the leader beyond its lease. Meanwhile, a client continues to write and call fsync</code>. The checker looks for overlapping commit epochs, lost durable writes, duplicates, stale reads, and a thawed leader that fails to recognize its fencing. The suite is in the repository</a>.</p> What it doesn't do</h2> HA improves availability, not throughput. There is still one writer, and reads are not distributed across nodes.</p> Crash failover takes roughly two seconds while the standby waits for the takeover TTL; clients block and retry during that interval. Advisory byte-range locks are not replicated and disappear on failover.</p> fsync</code> remains the durability boundary. A connected standby also protects unflushed writes, but applications should not rely on that extra copy. Configuration and a complete failure-case table are in the high availability documentation</a>.</p> Fencing makes the writer a storage-layer decision rather than a timing assumption. Replication covers ordinary failover; lineage tokens turn failures outside that coverage into ESTALE</code> instead of false success.</p>