Durability & Consistency

Transaction Scope

A write operation includes the following modifications within a single transaction:

• File data chunks
• Inode metadata (size, timestamps, mode)
• Directory entry updates
• Global statistics counters

The database commits these changes using a WriteBatch, which provides all-or-nothing semantics. Partial writes cannot occur.

Failure Modes

The intermediate state where some but not all modifications are visible is not possible.

Ordering Guarantee

The WriteCoordinator assigns monotonically increasing sequence numbers to operations. Each operation must wait for all predecessors to complete before committing:

let seq = write_coordinator.allocate_sequence();
seq.wait_for_predecessors().await;
db.write(txn).await;
seq.mark_committed();

This mechanism ensures that if operation A completes before operation B begins, then in any consistent state of the filesystem, the visibility of B implies the visibility of A.

Formal Property

For operations A and B where A completes before B starts:

visible(B) → visible(A)

The contrapositive also holds: if A is not visible, B cannot be visible.

Comparison with Other Filesystems

Many filesystems permit write reordering for performance optimization:

• ext4: May reorder writes within journal transactions
• XFS: Delayed allocation can reorder data relative to metadata
• Hardware: SSDs and disk controllers may reorder writes internally

Such reordering can result in states where a later write is visible while an earlier write is not. Applications that depend on ordering semantics (databases, logging systems, configuration management) may observe inconsistent state after recovery.

ZeroFS eliminates this class of problems by construction. The sequence number mechanism enforces a total order that survives crashes.

Recovery-Free Design

The underlying storage model has several properties that eliminate the need for crash recovery:

Atomic Commits: WriteBatch operations are atomic at the database level. A transaction either appears in full or not at all.

Immutable Storage: SST files written to object storage are immutable. Once uploaded, they cannot be corrupted by subsequent operations.

Self-Describing State: The database manifest describes the complete set of valid SST files. On startup, ZeroFS reads the manifest and has immediate access to the latest consistent state.

No Write-Ahead Log Dependencies: Unlike filesystems that require WAL replay to reconstruct state, ZeroFS state is fully materialized in SST files and the manifest.

Startup Procedure

After any termination (graceful or unexpected):

1. Read manifest from object storage
2. Load database state from referenced SST files
3. Resume operation

No scanning, replay, or repair is performed. The time to resume operation is independent of filesystem size or the nature of the previous termination.

Buffering Model

Write data flows through the following stages:

1. Application Buffer: User-space buffers managed by the application
2. Page Cache: OS-managed in-memory buffers
3. Memtable: In-memory LSM tree buffer
4. SST Files: Immutable files in object storage

Data moves from memtable to SST files when:

• The application calls fsync()
• The memtable reaches capacity
• The process terminates gracefully
• Periodic background flushes

POSIX Compliance

This behavior conforms to POSIX semantics, which specify that write() transfers data to system buffers while fsync() ensures data reaches stable storage. All major filesystems (ext4, XFS, APFS, ZFS...) implement this model.

Synchronization API

// POSIX interface
write(fd, data, len);   // Buffer in memory
fsync(fd);              // Persist to storage

# Python
f.write(data)
f.flush()
os.fsync(f.fileno())

// Go
f.Write(data)
f.Sync()

// Rust
file.write_all(data)?;
file.sync_all()?;

Applications requiring immediate durability must call the appropriate synchronization function.

9P Protocol

The 9P protocol provides direct mapping of POSIX synchronization semantics:

• fsync() system call generates a Tfsync protocol message
• ZeroFS flushes all buffered data to object storage
• The call returns only after durability is confirmed

This provides strong guarantees suitable for applications with strict durability requirements.

NFS Protocol

NFS client implementations do not reliably invoke the COMMIT operation:

• Clients typically report writes as stable without issuing COMMIT
• ZeroFS accepts this to avoid per-write latency penalties
• Effective durability depends on client behavior

For workloads requiring predictable durability semantics, the 9P protocol is recommended.

Failpoint Coverage

Failpoints are placed at critical points within each filesystem operation, allowing tests to simulate crashes at any stage:

Each failpoint triggers an immediate process termination, simulating power loss or crash at that exact point.

Consistency Verification

After each simulated crash, a comprehensive consistency checker validates the filesystem state:

verify_all()
  enumerate_inodes()
  enumerate_tombstones()
  walk_directory_tree()
  verify_directory_counts()
  verify_nlink_counts()
  verify_directory_nlinks()
  find_orphaned_inodes()
  verify_stats_counters()
  verify_tombstones()
  verify_file_chunks()
  verify_inode_counter()
  verify_orphaned_chunks()
  verify_dir_entry_scan_consistency()
  verify_orphaned_directory_metadata()
  verify_dir_cookie_counters()

The checker detects: dangling references, orphaned inodes, nlink mismatches, missing chunks, stale tombstones, counter inconsistencies, and directory entry corruption.

POSIX Compliance

pjdfstest: A POSIX filesystem compliance test suite originally developed for FreeBSD. Tests cover file creation, permissions, hard links, symbolic links, timestamps, and other POSIX semantics. Runs on both NFS and 9P protocols.

xfstests: The Linux kernel's filesystem test suite, originally developed for XFS but now used across all major Linux filesystems. Provides extensive coverage of edge cases, concurrent operations, and error handling.

Stress Testing

stress-ng: Exercises filesystem operations under concurrent load, including directory operations, file metadata, links, renames, and attribute modifications. Tests run for sustained periods on both NFS and 9P.

Linux Kernel Compilation: Compiles the Linux kernel source tree on ZeroFS, exercising real-world workload patterns including parallel file creation, compilation, and linking across thousands of files.