Configuration

ZeroFS reads storage, cache, server, and runtime settings from a TOML file passed to zerofs run.

Create a Configuration

Generate the default template, then edit it for the storage backend and listeners you intend to use:

zerofs init

# Cache configuration (required)
[cache]
dir = "/var/cache/zerofs"        # Directory for caching data
disk_size_gb = 10.0              # Maximum disk cache size in GB
memory_size_gb = 2.0             # Memory cache size in GB (optional)

# Storage configuration (required)
[storage]
url = "s3://bucket/path"         # Storage backend URL
encryption_password = "your-secure-password"  # Encryption password

# Server configuration (required table, every subsection optional)
[servers.nfs]
addresses = ["127.0.0.1:2049"]

Three tables are required: [cache], [storage], and [servers]. A file missing any of them fails to parse; omitting [servers] reports missing field servers. Each [servers.*] subsection is optional. An empty [servers] table parses, but zerofs run then exits with No servers configured. At least one server (NFS, 9P, NBD, or RPC) must be enabled. At least one subsection must start a listener: for 9P, NBD, and RPC a unix_socket alone counts, and a [servers.webui] section counts in Web-UI-enabled builds.

Two parsing rules apply to the whole file:

• Unknown keys are rejected. A misspelled key aborts startup with a parse error naming the key. The exception is the free-form backend tables [aws], [azure], and [gcp]; see Backend Option Tables.
• Sizes are decimal gigabytes. All *_gb keys (cache.disk_size_gb, cache.memory_size_gb, filesystem.max_size_gb) are interpreted as 10⁹ bytes, not GiB. Fractional values are accepted: 0.5 is 500 MB.

Cache

The [cache] table is required. ZeroFS runs two caches, each with a memory and a disk tier: a decoded-block cache for metadata and a raw-parts cache for object bytes. Both disk tiers live under dir, inside a per-bucket subdirectory: bucket_<8hex>/hybrid_cache/ and bucket_<8hex>/parts_cache/; see Caching for the layout. File data in the parts cache is stored as fetched from the object store: compressed and encrypted.

See Caching for how the two budgets are split between the tiers.

Storage Backends

URL Schemes

The url field under [storage] accepts the following schemes.

az:// and adl:// also parse as Azure aliases. adl://container/path and the bare abfs://container/path form behave like azure://container/path. In the az:// form, the host is used as the container name and the first path segment is excluded from the object path; use azure:// instead.

Full https:// URLs are routed by host suffix. Hosts ending in amazonaws.com or r2.cloudflarestorage.com are treated as S3 (https://bucket.s3.region.amazonaws.com, https://ACCOUNT_ID.r2.cloudflarestorage.com/bucket). Hosts ending in dfs.core.windows.net, blob.core.windows.net, dfs.fabric.microsoft.com, or blob.fabric.microsoft.com are treated as Azure (https://account.blob.core.windows.net/container/path). Any other http:// or https:// URL is rejected at startup with feature for Http not enabled; generic HTTP and WebDAV endpoints are not supported. For S3-compatible services on other hosts, such as MinIO, use url = "s3://bucket/path" with endpoint under [aws], plus allow_http = "true" for plain-HTTP endpoints.

A scheme selects the backend only. Credentials come from the [aws], [azure], or [gcp] table regardless of which alias is used.

AWS S3

[storage]
url = "s3://my-bucket/zerofs-data"
encryption_password = "secure-password-here"
# storage_class = "..."          # Optional, provider-specific; see Storage Class below

[aws]
access_key_id = "AKIAIOSFODNN7EXAMPLE"
secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region = "us-east-1"              # Optional, defaults to us-east-1
endpoint = "https://s3.amazonaws.com"  # Optional for S3-compatible services
allow_http = "false"                # Set to "true" (quoted) for plain-HTTP endpoints
conditional_put = "redis://localhost:6379"  # For stores without conditional put support

Conditional Put Support

ZeroFS requires conditional write (put-if-not-exists) support for fencing. AWS S3 supports this natively, so the conditional_put option is not needed when using AWS S3 directly.

For S3-compatible object stores that do not support conditional puts, set conditional_put to a Redis URL. ZeroFS will use Redis to coordinate conditional write operations:

[aws]
conditional_put = "redis://localhost:6379"

This is required for object stores that return success instead of an error when a put-if-not-exists operation targets an existing object.

Azure Blob Storage

[storage]
url = "azure://container/path"
encryption_password = "secure-password-here"

[azure]
storage_account_name = "myaccount"
storage_account_key = "your-account-key"

Google Cloud Storage (GCS)

Using Application Default Credentials (GCP VMs/GKE):

[storage]
url = "gs://my-bucket/zerofs-data"
encryption_password = "secure-password-here"

# No [gcp] section needed when running on GCP with attached service account

Using Service Account Key File:

[storage]
url = "gs://my-bucket/zerofs-data"
encryption_password = "secure-password-here"

[gcp]
service_account = "/path/to/service-account-key.json"
# Or use application_credentials = "${GOOGLE_APPLICATION_CREDENTIALS}"

Backend Option Tables

[aws], [azure], and [gcp] are free-form key-value tables, not fixed schemas. At startup, each key is lowercased, prefixed with aws_, azure_, or google_, and forwarded to the object-store builder. For example, [aws] accepts session_token, virtual_hosted_style_request, and skip_signature; Azure's storage_sas_token is forwarded as azure_storage_sas_token.

The object-store documentation lists the accepted keys: AmazonS3ConfigKey, AzureConfigKey, and GoogleConfigKey.

Two rules apply:

• Every value must be a TOML string. allow_http = "true", not allow_http = true. A non-string value fails at startup with Failed to parse config file.
• Unknown keys are dropped without warning. A misspelled key such as secret_acess_key produces no configuration error and surfaces later as an authentication or connection failure. Check key spelling before debugging credentials.

Values in all of these tables support environment variable substitution.

Storage Class

The optional storage_class field under [storage] sets the object storage class/tier for every write. The value is sent verbatim as the backend's own tiering header on every PUT and multipart upload: x-amz-storage-class (S3), x-goog-storage-class (GCS), x-ms-access-tier (Azure).

[storage]
url = "s3://my-bucket/zerofs-data"
encryption_password = "secure-password-here"
storage_class = "..."  # provider-specific value, e.g. a single-zone hot class

Because the value is verbatim, it must be valid for your backend; a value the backend does not recognize (an S3 class name on Azure, for example) is rejected by the store on the first write. Omit the field to use the account or container default.

Server-side copies cannot carry a storage class: objects written via copy land in the bucket or account default class regardless of this setting.

Network Services

The [servers] table is required; each subsection below is optional. A server runs only when its section is present. An empty [servers] table is not runnable: zerofs run exits with No servers configured. At least one server (NFS, 9P, NBD, or RPC) must be enabled.

addresses is a list of socket addresses. Each entry binds one listener, so multiple entries serve the same filesystem on multiple addresses. IPv6 addresses use brackets:

[servers.nfs]
addresses = ["0.0.0.0:2049"]                    # All IPv4 interfaces
# addresses = ["[::]:2049"]                     # All IPv6 interfaces
# addresses = ["127.0.0.1:2049", "[::1]:2049"]  # Dual-stack localhost

Omitting addresses starts no TCP listener; there is no implicit 127.0.0.1 bind. The 127.0.0.1 addresses for NFS (2049), 9P (5564), NBD (10809), and RPC (7000) appear in the template generated by zerofs init, not as built-in defaults. The 9P, NBD, and RPC servers can serve over unix_socket alone.

NFS Server

[servers.nfs]
addresses = ["0.0.0.0:2049"]

NFS is TCP-only: [servers.nfs] accepts only addresses and has no unix_socket key.

9P Server

[servers.ninep]
addresses = ["0.0.0.0:5564"]
unix_socket = "/tmp/zerofs.9p.sock"  # Optional

NBD Server

[servers.nbd]
addresses = ["0.0.0.0:10809"]
unix_socket = "/tmp/zerofs.nbd.sock"  # Optional

RPC Server

[servers.rpc]
addresses = ["127.0.0.1:7000"]
unix_socket = "/tmp/zerofs.rpc.sock"

The admin endpoint for zerofs checkpoint, zerofs flush, zerofs monitor, zerofs fatrace, and zerofs otrace. These commands read the same configuration file, try the Unix socket first if the socket file exists, then each configured address in turn (the order is not defined). Without a [servers.rpc] section, they exit with RPC server not configured in config file. See Monitoring and Checkpoints.

Web UI

[servers.webui]
addresses = ["127.0.0.1:8080"]  # Default when the section is present
uid = 1000                      # Required
gid = 1000                      # Required

uid and gid set the POSIX identity for all file operations performed from the browser. See Web UI.

Metrics and Telemetry

The [prometheus] and [telemetry] tables sit at the top level, not under [servers]:

[prometheus]
addresses = ["127.0.0.1:9091"]  # Default when the section is present

[telemetry]
enabled = true  # Default, also when the section is absent

Without a [prometheus] section, no metrics endpoint is started. Telemetry defaults to enabled whether or not the section is present. See Prometheus Metrics and Telemetry.

Filesystem Quotas

ZeroFS supports configurable filesystem size limits:

[filesystem]
max_size_gb = 100.0  # Limit filesystem to 100 GB

When the quota is reached, write operations return ENOSPC (No space left on device). Delete and truncate operations continue to work, allowing you to free space. If not specified, the limit defaults to 16 EiB, the maximum filesystem size.

Compression

ZeroFS compresses each 32 KiB extent of file data before encrypting it; the resulting frames are packed into segment objects on the object store. The default is tuned for object storage, where network and storage cost dominate local CPU:

[filesystem]
compression = "zstd-3"   # Zstd at level 3 (default)
# or
compression = "lz4"      # Fast compression, lower ratio

Algorithms

• zstd-{level} (default: zstd-3): Zstandard compression with configurable level from 1 to 22. Lower levels (1-5) are faster, higher levels (15-22) achieve better compression but are slower.

• lz4: Very fast compression and decompression with moderate compression ratio. Prefer for write-throughput-bound workloads where compression CPU is the bottleneck.

Changing Compression

You can change the compression algorithm at any time without migration:

• New writes use the currently configured algorithm
• Existing data remains readable regardless of how it was compressed
• Compression format is auto-detected per-frame when reading
• Existing data is never re-encoded: repacking relocates frames byte-for-byte without recompressing, so changing the algorithm or level does not rewrite or shrink data already stored; old and new encodings coexist indefinitely until the data is overwritten or removed

LSM Tree Performance Tuning

ZeroFS keeps metadata (inodes, directory entries, and the per-extent frame pointers) in an LSM (Log-Structured Merge) tree database that sits in the object store. File contents live in immutable segment objects written directly to the object store; the [reclaim] section below tunes how their space is reclaimed. The [lsm] section tunes the metadata store, though flush_interval_secs and sync_writes also govern when buffered file data is sealed and uploaded. See Architecture.

[lsm]
l0_max_ssts = 256                # Max SST files in L0 before compaction
max_concurrent_compactions = 64  # Max concurrent compaction operations
flush_interval_secs = 30         # Interval between periodic flushes (in seconds)
sync_writes = false              # Make every write durable on return

Parameters

• l0_max_ssts (default: 256, min: 4): Maximum number of SST (Sorted String Table) files allowed in level 0 before triggering compaction. The same value applies per key. Lower values reduce read amplification but increase write amplification. Higher values do the opposite.

• max_concurrent_compactions (default: 64, min: 1): Maximum number of metadata compaction operations that can run concurrently. This sets both the coordinator's scheduling limit and the embedded worker's execution limit. The scheduler starts jobs as eligible work becomes available, up to this limit. A higher limit lets independent compactions drain a backlog concurrently; lower it to reduce background CPU, memory, and object-store request use. Compaction always runs embedded in zerofs run; there is no separate compactor process.

• flush_interval_secs (default: 30, min: 5): Interval in seconds between periodic background flushes to durable storage. Each flush first seals the open segment buffer, uploading buffered file data, then flushes the metadata store. Between flushes, up to 256 MiB of committed writes can sit in the in-memory open segment. fsync calls from clients (9P fsync, NFS COMMIT, NBD flush) always trigger an immediate flush regardless of this interval, unless [filesystem] ignore_fsync = true, which makes fsync a no-op; see High Availability.

• sync_writes (default: false): When enabled, every write is durably flushed before returning success, eliminating the "in-memory unflushed" window between fsync calls. This does not change POSIX fsync semantics: with the default false, explicit fsync from clients is still honored and waits for durable persistence. The flag only governs writes between fsync calls. Expensive: each coalesced batch forces a segment seal plus a memtable flush (concurrent writes still merge into a single batch and a single flush). Rejected in combination with [filesystem] ignore_fsync. See the Eliminating the Durability Window section for guidance on when to enable.

When to Tune LSM Parameters

Consider tuning these parameters if you're experiencing:

• High read latency: Decrease l0_max_ssts to reduce read amplification
• CPU / Network underutilization: Increase max_concurrent_compactions on high-core systems
• Too-frequent background flushes: Increase flush_interval_secs; the crash-loss window for un-fsync'd writes grows accordingly

Segment Reclamation

The [reclaim] section tunes the segment-reclamation loop described under Space Reclamation, not tombstone cleanup, the orphan sweep, or metadata compaction.

[reclaim]
max_concurrent_repacks = 4       # Independent segment-repack jobs (range 1-16)
repack_min_dead_percent = 10     # Repack non-small segments above this % dead (range 1-99)

Parameters

• max_concurrent_repacks (default: 4, range: 1–16): Maximum number of independent source sets gathered, packed, PUT, and repointed at once. Foreground traffic does not reduce this job limit. Each job can issue bounded payload I/O, while metadata point work shares a separate global bound. Lower it to reduce object-store fan-out and memory reservations; raise it when the backend and host can sustain more parallel work.

• repack_min_dead_percent (default: 10, range: 1–99): Minimum dead percentage for repacking a segment at least 1 MiB large. Smaller segments remain candidates for consolidation. Lower values recover more space but rewrite more live data; raise it on request-billed or rewrite-sensitive backends.

Each job uses a fixed nominal selection and gather budget of 256 MiB of stored live frame data. At the default concurrency of four jobs, the aggregate budget is 1 GiB. A job may exceed its soft budget for the first source to ensure progress; the complete encoded output is limited to 1 GiB. Buffers and metadata consume additional memory. Segment uploads use one PUT rather than multipart.

Multiple Instances

One read-write instance and any number of read-only instances can run on the same storage backend, all using the same configuration file. See Read Replicas for startup, freshness, and restrictions.

High Availability

A [replication] section runs the node as one half of a leader/standby pair over a single object store, with automatic failover that loses no fsync'd data. Each node has its own config file. See High Availability for the design, guarantees, and a full two-node example.

[replication]
node_id = "node-a"
role = "leader"                       # initial role: "leader" or "standby"
replication_listen = "10.0.0.1:9000"  # this node receives ships and heartbeats here
peers = ["10.0.0.2:9000"]             # the other node's replication_listen

Set both replication_listen and peers on every node so either can lead after a role swap. ZeroFS rejects a configuration that sets only one. A standalone leader may omit both.

While the pair is Connected, the leader ships writes before acknowledging them, so unflushed acknowledged writes survive one node failure. In Solo mode, they have the standalone durability window. Setting [filesystem] ignore_fsync = true makes fsync a no-op and provides neither an object-store barrier nor a lineage check.

S3-Compatible Examples

[cache]
dir = "/var/cache/zerofs"
disk_size_gb = 10.0

[storage]
url = "s3://my-bucket/path"
encryption_password = "secure-password"

[aws]
access_key_id = "minioadmin"
secret_access_key = "minioadmin"
endpoint = "https://minio.example.com"
allow_http = "true"  # Quoted string "true", not a bare boolean

[servers.nfs]
addresses = ["0.0.0.0:2049"]

Running ZeroFS

With Configuration File

# Start ZeroFS with a config file
zerofs run --config /etc/zerofs/zerofs.toml

# Or use the shorthand
zerofs run -c zerofs.toml

Password Management

To change the encryption password:

# Reads the new password from stdin: type one line and press Enter, or pipe it
zerofs change-password --config zerofs.toml
echo "newpassword" | zerofs change-password -c zerofs.toml

The command reads exactly one line from stdin and re-wraps the stored encryption key with the new password. File data is not re-encrypted.

Environment Variable Substitution

Configuration values can reference environment variables using $VAR or ${VAR} syntax:

[storage]
url = "s3://my-bucket/data"
encryption_password = "${ZEROFS_PASSWORD}"

[aws]
access_key_id = "${AWS_ACCESS_KEY_ID}"
secret_access_key = "${AWS_SECRET_ACCESS_KEY}"

Substitution applies to these keys only; all other values are taken literally:

• url, encryption_password, and storage_class under [storage]
• dir under [cache]
• unix_socket under [servers.ninep], [servers.nbd], and [servers.rpc]
• addresses under [servers.nfs], [servers.ninep], [servers.nbd], [servers.rpc], [servers.webui], and [prometheus]
• every value in [aws], [azure], and [gcp]
• node_id, replication_listen, and peers under [replication]

For array values (peers and the addresses lists) each entry is expanded independently, so peers = ["${PEER_A}", "${PEER_B}"] substitutes each element on its own; a single variable is not split into multiple entries. Expansion runs on the whole string before an addresses entry is parsed as a socket address, so the host, the port, or the entire value may come from a variable: addresses = ["${POD_IP}:2049"] and addresses = ["${BIND_ADDR}"] are both valid.

If a referenced variable is unset, the configuration fails to load with Failed to expand environment variable. A literal $ in an expandable value (an encryption password, for example) must be written $$.

This keeps secrets out of configuration files and lets the same file serve several environments.

System Integration

systemd Service

Create /etc/systemd/system/zerofs.service:

[Unit]
Description=ZeroFS filesystem
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/zerofs run --config /etc/zerofs/zerofs.toml
Restart=always
RestartSec=5
# Optional: Load environment variables for substitution
EnvironmentFile=-/etc/zerofs/zerofs.env

[Install]
WantedBy=multi-user.target

If using environment variable substitution, create /etc/zerofs/zerofs.env:

ZEROFS_PASSWORD=your-secure-password
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret

Docker

# Create config file
cat > zerofs.toml <<EOF
[cache]
dir = "/cache"
disk_size_gb = 10.0

[storage]
url = "s3://bucket/path"
encryption_password = "\${ZEROFS_PASSWORD}"

[aws]
access_key_id = "\${AWS_ACCESS_KEY_ID}"
secret_access_key = "\${AWS_SECRET_ACCESS_KEY}"

[servers.nfs]
addresses = ["0.0.0.0:2049"]
EOF

# Run container
docker run -d \
  -e ZEROFS_PASSWORD='secure-password' \
  -e AWS_ACCESS_KEY_ID='your-key' \
  -e AWS_SECRET_ACCESS_KEY='your-secret' \
  -v $(pwd)/zerofs.toml:/config/zerofs.toml:ro \
  -v /tmp/cache:/cache \
  -p 2049:2049 \
  ghcr.io/barre/zerofs:latest run --config /config/zerofs.toml

Configuration files can contain the encryption password and storage credentials. Restrict their file permissions and do not commit secrets to version control. Environment-variable substitution is available when the deployment's secret-management mechanism supplies values that way.

Logging Configuration

Set log levels using the RUST_LOG environment variable. With RUST_LOG unset, zerofs run logs at info for all crates; zerofs mount uses info,fuser::reply=off. Logs are written to stderr.

RUST_LOG=zerofs=debug zerofs run -c zerofs.toml