Storage Engine: Segments and Extents

ZeroFS stores file contents in immutable segment objects written directly to the object store. An LSM tree on the same object store holds only metadata: inodes, directory entries, and one 32-byte pointer per extent.

Overview

File data is split into 32 KiB extents. Each extent is compressed, then encrypted into a self-describing frame. Frames are packed into segments: immutable objects written once to the object store and removed only by segment reclamation. The two stores divide the work:

Segment object keys are segments/{shard}/{epoch}/{counter}:

• epoch: the writer epoch, a fencing counter bumped on every writer open. Two writer terms can never collide on an object key.
• counter: a per-process sequence number, starting at 0 each open.
• shard: the low byte of the counter, as two hex digits. Consecutive segments round-robin across 256 prefixes, avoiding object-store hot-prefix throttling on a monotonic key tail. Reads resolve exact keys from the FrameLoc, so sharding costs the read path nothing.

A write touches only RAM and the metadata transaction. Object-store requests happen at segment seal and metadata flush, not per write.

On-Object Format

A segment has three parts, written in order:

[frame_0][frame_1]...[frame_k-1]   packed, no padding
[directory]                        AEAD-sealed reverse map (reclamation and recovery only)
[footer]                           64 bytes, plaintext, written last

Each frame on the wire is [len: u32 LE][sealed_frame]. The sealed frame is [nonce: 24 bytes][ciphertext + tag: 16 bytes]: the extent is compressed (per the compression setting), then encrypted with XChaCha20-Poly1305. The key is derived from the master key via HKDF-SHA256 with the label zerofs-v1-segment, domain-separated from the metadata SST-block key. The AEAD associated data binds each frame to its segment ID, frame index, inode, and extent index; a frame cannot be moved to another segment, slot, or logical block without failing authentication.

The FrameLoc is the LSM value of an extent key: 32 bytes, little-endian.

The footer is 64 bytes, fixed layout:

The CRC32C is a keyless torn-write detector for the directory and footer only; frames carry their own AEAD tags. The directory is a reverse map, one 28-byte entry per frame (byte offset, length, inode, extent index), sealed as a single AEAD frame. Normal reads never touch it; only segment reclamation, repacking, and standby recovery do.

Write Path

1. A file write splits into 32 KiB extents. Partially overwritten extents are read back first, up to 20 in parallel; full overwrites and writes beyond EOF skip the read. All-zero extents become key deletions (holes).
2. Each non-zero extent is sealed (compressed, encrypted) and appended to an in-RAM open segment buffer. The FrameLoc put and the segment live-byte credits and debits commit in the same transaction as the inode update. No object-store request is issued on the write path.
3. When the open buffer crosses 256 MiB, it rotates and uploads in the background as one PUT. At most 4 seals run in flight; this bounds un-uploaded segment RAM to 4 × 256 MiB and backpressures writers. A failed background upload is retried at the next flush.

The fsync barrier

fsync (and every other flush) runs a fixed sequence: drain in-flight seals, re-upload any failed seal, seal the current open buffer, then flush the metadata LSM. If sealing fails, the metadata flush does not run. The invariant: a durable metadata manifest never references an un-uploaded segment. A crash between segment upload and metadata flush leaves an orphaned segment object, which the orphan sweep reclaims, never a durable pointer to a missing object. This invariant is exercised by failpoint crash tests.

Metadata becomes durable only through this coordinated barrier, so no metadata persistence path can bypass segment upload.

Flushes are triggered by:

• Client fsync, NFS COMMIT, NBD flush
• The periodic flush every flush_interval_secs (default 30 s)
• Every commit batch when sync_writes = true
• Each segment-reclamation cycle
• Shutdown

Between flushes, up to 256 MiB of committed writes exist only in the open RAM buffer, plus up to 4 × 256 MiB in sealing buffers. Under replication, the standby also holds these writes: a frame still in leader RAM ships with the replicated operation, and a promoted standby rebuilds any missing segment objects at takeover. See Durability & Consistency for the guarantee model.

Read Path

A point read looks up the extent key, decodes the FrameLoc, and checks the open buffer and sealing map first (read-your-writes without a network round trip). On a miss it issues one ranged GET of exactly the frame's byte span, then decrypts and decompresses. Frames are self-describing; the segment directory is never consulted on reads.

Multi-extent reads scan the extent keyspace and coalesce every maximal run that is contiguous in one segment (same segment, consecutive frame indices, adjacent byte ranges) into a single ranged GET. Missing extents are holes and return zeros. Segment repacking re-sorts a file's live extents by (inode, extent index), so a whole-file read of a packed file can be one GET.

Two prefetch layers apply:

• Logical read-ahead: after 2 consecutive sequential reads, an 8 MiB window of upcoming extents is prefetched, up to 16 fetches in parallel. It fires only when the window crosses into a different segment object; within one segment the physical layer covers it.
• Physical prefetch: objects are cached in 128 KiB parts; a sequential stream's fetch window ramps from 128 KiB to 8 MiB, kept up to 4 windows deep. Concurrent misses on the same window share one GET.

The parts cache stores raw object bytes: extent data is compressed and encrypted on the local cache disk. Just-sealed segments are inserted into the parts cache from the bytes in hand, so a read after write is served locally. Cache sizing is documented on the Caching page.

Space Reclamation

The metadata LSM tracks live bytes per segment. Transactions carry signed deltas; the single-writer commit worker folds them into absolute counters. Overwriting or deleting an extent debits the segment that held its old frame.

A continuous segment-reclamation task runs only on the writer. It performs these steps:

1. Seal and flush under the barrier, so the durable view matches memory.
2. Scan the per-segment counters: O(number of segments), not extents. Reclaim scans do not list the object store; the only listing is the daily orphan sweep.
3. Delete dead segments in parallel after a safety horizon and a check that no extent still references them. A read or decode error keeps the segment. Checkpoints extend or pause deletion.
4. Repack dead-space or small-segment candidates in parallel into larger objects, then conditionally repoint extents that still reference the source frames. Up to four independent jobs run at once by default.

The task feeds each complete candidate snapshot to bounded executors until no economic repack selection remains, an error occurs, or shutdown begins. It scans once at startup, then only when a committed counter update creates dead bytes, a sub-1-MiB segment is sealed, or a pending deletion reaches its deadline. Activity is coalesced to the configured flush cadence; an idle store performs no counter scans. The Space Reclamation page documents concurrency, thresholds, request costs, crash safety, checkpoint interaction, and metrics. File data does not participate in LSM compaction; the embedded compactor rewrites metadata SSTs only.

Design Constants

These values are compile-time constants:

Configuration

The engine responds to these configuration keys:

Cache sizing ([cache] disk_size_gb, memory_size_gb, warm_metadata) is covered on the Caching page.