Configuration Guide

ZeroFS uses TOML configuration files for all settings. This guide covers all available configuration options.

Getting Started

Create a configuration file using the init command:

zerofs init

This creates a template configuration file that you can customize.

Configuration File Structure

Basic Configuration

# Cache configuration (required)
[cache]
dir = "/var/cache/zerofs"        # Directory for caching data
disk_size_gb = 10.0              # Maximum disk cache size in GB
memory_size_gb = 2.0             # Memory cache size in GB (optional)

# Storage configuration (required)
[storage]
url = "s3://bucket/path"         # Storage backend URL
encryption_password = "your-secure-password"  # Encryption password

# Server configuration (required table, every subsection optional)
[servers.nfs]
addresses = ["127.0.0.1:2049"]

Three tables are required: [cache], [storage], and [servers]. A file missing any of them fails to parse — omitting [servers] reports missing field servers. Each [servers.*] subsection is optional, and an empty [servers] table is valid: it parses and starts no servers.

Two parsing rules apply to the whole file:

• Unknown keys are rejected. A misspelled key aborts startup with a parse error naming the key. The exception is the free-form backend tables [aws], [azure], and [gcp] — see Backend Option Tables.
• Sizes are decimal gigabytes. All *_gb keys (cache.disk_size_gb, cache.memory_size_gb, filesystem.max_size_gb, lsm.max_unflushed_gb) are interpreted as 10⁹ bytes, not GiB. Fractional values are accepted: 0.5 is 500 MB.

Storage Backends

URL Schemes

The url field under [storage] accepts the following schemes. The same parser handles [wal].url.

az:// and adl:// also parse as Azure aliases. adl://container/path and the bare abfs://container/path form behave like azure://container/path. In the az:// form, the host is used as the container name and the first path segment is excluded from the object path; use azure:// instead.

Full https:// URLs are routed by host suffix. Hosts ending in amazonaws.com or r2.cloudflarestorage.com are treated as S3 (https://bucket.s3.region.amazonaws.com, https://ACCOUNT_ID.r2.cloudflarestorage.com/bucket). Hosts ending in dfs.core.windows.net, blob.core.windows.net, dfs.fabric.microsoft.com, or blob.fabric.microsoft.com are treated as Azure (https://account.blob.core.windows.net/container/path). Any other http:// or https:// URL is rejected at startup with feature for Http not enabled; generic HTTP and WebDAV endpoints are not supported. For S3-compatible services on other hosts, such as MinIO, use url = "s3://bucket/path" with endpoint under [aws], plus allow_http = "true" for plain-HTTP endpoints.

A scheme selects the backend only. Credentials come from the [aws], [azure], or [gcp] table regardless of which alias is used.

AWS S3

[storage]
url = "s3://my-bucket/zerofs-data"
encryption_password = "secure-password-here"
# storage_class = "..."          # Optional, provider-specific; see Storage Class below

[aws]
access_key_id = "AKIAIOSFODNN7EXAMPLE"
secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region = "us-east-1"              # Optional, defaults to us-east-1
endpoint = "https://s3.amazonaws.com"  # Optional for S3-compatible services
allow_http = "false"                # Set to "true" (quoted) for plain-HTTP endpoints
conditional_put = "redis://localhost:6379"  # For stores without conditional put support

Conditional Put Support

ZeroFS requires conditional write (put-if-not-exists) support for fencing. AWS S3 supports this natively, so the conditional_put option is not needed when using AWS S3 directly.

For S3-compatible object stores that do not support conditional puts, set conditional_put to a Redis URL. ZeroFS will use Redis to coordinate conditional write operations:

[aws]
conditional_put = "redis://localhost:6379"

This is required for object stores that return success instead of an error when a put-if-not-exists operation targets an existing object.

Azure Blob Storage

[storage]
url = "azure://container/path"
encryption_password = "secure-password-here"

[azure]
storage_account_name = "myaccount"
storage_account_key = "your-account-key"

Google Cloud Storage (GCS)

Using Application Default Credentials (GCP VMs/GKE):

[storage]
url = "gs://my-bucket/zerofs-data"
encryption_password = "secure-password-here"

# No [gcp] section needed when running on GCP with attached service account

Using Service Account Key File:

[storage]
url = "gs://my-bucket/zerofs-data"
encryption_password = "secure-password-here"

[gcp]
service_account = "/path/to/service-account-key.json"
# Or use application_credentials = "${GOOGLE_APPLICATION_CREDENTIALS}"

Local Filesystem

[storage]
url = "file:///path/to/storage"
encryption_password = "secure-password-here"

# No additional backend configuration needed

Backend Option Tables

[aws], [azure], and [gcp] are free-form key-value tables, not fixed schemas. At startup, each key is lowercased, prefixed with aws_, azure_, or google_, and forwarded to the object_store builder for the storage URL's backend. Any configuration key that builder accepts works — for example session_token, virtual_hosted_style_request, or skip_signature under [aws], and sas_token under [azure]. The full key lists are in the object_store documentation: AmazonS3ConfigKey, AzureConfigKey, GoogleConfigKey.

Two rules apply:

• Every value must be a TOML string. allow_http = "true", not allow_http = true. A non-string value fails at startup with Failed to parse config file.
• Unknown keys are dropped without warning. A misspelled key — secret_acess_key — produces no configuration error and surfaces later as an authentication or connection failure. Check key spelling before debugging credentials.

The same rules apply to the [wal.aws], [wal.azure], and [wal.gcp] tables used with a separate WAL store. Values in all of these tables support environment variable substitution.

Storage Class

The optional storage_class field under [storage] sets the object storage class/tier for every write. The value is sent verbatim as the backend's own tiering header on every PUT and multipart upload: x-amz-storage-class (S3), x-goog-storage-class (GCS), x-ms-access-tier (Azure).

[storage]
url = "s3://my-bucket/zerofs-data"
encryption_password = "secure-password-here"
storage_class = "..."  # provider-specific value, e.g. a single-zone hot class

Because the value is verbatim, it must be valid for your backend; a value the backend does not recognize — an S3 class name on Azure, for example — is rejected by the store on the first write. Omit the field to use the account/bucket default. The typical use is selecting a cheaper single-zone / reduced-redundancy hot or higher performance class where your provider offers one.

Server-side copies cannot carry a storage class: objects written via copy land in the bucket or account default class regardless of this setting. [wal] has its own independent storage_class and does not inherit this one — see Separate WAL Object Store.

Network Services

The [servers] table is required; each subsection below is optional. A server runs only when its section is present, and an empty [servers] table starts no servers.

addresses is a list of socket addresses. Each entry binds one listener, so multiple entries serve the same filesystem on multiple addresses. IPv6 addresses use brackets:

[servers.nfs]
addresses = ["0.0.0.0:2049"]                    # All IPv4 interfaces
# addresses = ["[::]:2049"]                     # All IPv6 interfaces
# addresses = ["127.0.0.1:2049", "[::1]:2049"]  # Dual-stack localhost

Omitting addresses starts no TCP listener; there is no implicit 127.0.0.1 bind. The 127.0.0.1 addresses for NFS (2049), 9P (5564), NBD (10809), and RPC (7000) appear in the template generated by zerofs init, not as built-in defaults. The 9P, NBD, and RPC servers can serve over unix_socket alone.

NFS Server

[servers.nfs]
addresses = ["0.0.0.0:2049"]

NFS is TCP-only: [servers.nfs] accepts only addresses and has no unix_socket key.

9P Server

[servers.ninep]
addresses = ["0.0.0.0:5564"]
unix_socket = "/tmp/zerofs.9p.sock"  # Optional

NBD Server

[servers.nbd]
addresses = ["0.0.0.0:10809"]
unix_socket = "/tmp/zerofs.nbd.sock"  # Optional

RPC Server

[servers.rpc]
addresses = ["127.0.0.1:7000"]
unix_socket = "/tmp/zerofs.rpc.sock"

The admin endpoint for zerofs checkpoint, zerofs flush, zerofs monitor, and zerofs fatrace. These commands read the same configuration file, try the Unix socket first if the socket file exists, then each address in order. Without a [servers.rpc] section, they exit with RPC server not configured in config file. See Monitoring and Checkpoints.

Web UI

[servers.webui]
addresses = ["127.0.0.1:8080"]  # Default when the section is present
uid = 1000                      # Required
gid = 1000                      # Required

uid and gid set the POSIX identity for all file operations performed from the browser. See Web UI.

Metrics and Telemetry

The [prometheus] and [telemetry] tables sit at the top level, not under [servers]:

[prometheus]
addresses = ["127.0.0.1:9091"]  # Default when the section is present

[telemetry]
enabled = true  # Default, also when the section is absent

Without a [prometheus] section, no metrics endpoint is started. Telemetry defaults to enabled whether or not the section is present. See Prometheus Metrics and Telemetry.

Filesystem Quotas

ZeroFS supports configurable filesystem size limits:

[filesystem]
max_size_gb = 100.0  # Limit filesystem to 100 GB

When the quota is reached, write operations return ENOSPC (No space left on device). Delete and truncate operations continue to work, allowing you to free space. If not specified, the limit defaults to 16 EiB, the maximum filesystem size.

Compression

ZeroFS compresses file data (chunks) before encryption. The default is tuned for object storage, where network and storage cost dominate local CPU:

[filesystem]
compression = "zstd-3"   # Zstd at level 3 (default)
# or
compression = "lz4"      # Fast compression, lower ratio

Algorithms

• zstd-{level} (default: zstd-3): Zstandard compression with configurable level from 1 to 22. Lower levels (1-5) are faster, higher levels (15-22) achieve better compression but are slower.

• lz4: Very fast compression and decompression with moderate compression ratio. Prefer for write-throughput-bound workloads where compression CPU is the bottleneck.

Changing Compression On-the-fly

You can change the compression algorithm at any time without migration:

• New writes use the currently configured algorithm
• Existing data remains readable regardless of how it was compressed
• Compression format is auto-detected per-chunk when reading

LSM Tree Performance Tuning

ZeroFS uses an LSM (Log-Structured Merge) tree as its underlying storage engine. For advanced performance tuning, you can configure LSM parameters:

[lsm]
l0_max_ssts = 64                 # Max SST files in L0 before compaction
max_unflushed_gb = 1.0           # Max unflushed data before forcing flush (in GB)
max_concurrent_compactions = 8   # Max concurrent compaction operations
flush_interval_secs = 30         # Interval between periodic flushes (in seconds)
wal_enabled = false              # Whether the write-ahead log is enabled
sync_writes = false              # Make every write durable on return (HA mode)

Parameters

• l0_max_ssts (default: 64, min: 4): Maximum number of SST (Sorted String Table) files allowed in level 0 before triggering compaction. Lower values reduce read amplification but increase write amplification. Higher values do the opposite.

• max_unflushed_gb (default: 1.0, min: 0.1): Maximum amount of unflushed data (in gigabytes) before forcing a flush to storage. Supports fractional values like 0.5 for 500 MB. Larger values improve write batching but increase memory usage.

• max_concurrent_compactions (default: 8, min: 1): Maximum number of compaction operations that can run concurrently. Higher values can improve throughput on systems with many CPU cores and high network throughput but increase memory usage.

• flush_interval_secs (default: 30, min: 5): Interval in seconds between periodic background flushes to durable storage. Lower values flush more frequently. Higher values improve write throughput by batching more data before flushing. Note that fsync calls on NBD and 9P always trigger an immediate flush regardless of this interval.

• wal_enabled (default: false): Controls the write-ahead log (WAL). Can be changed at any time without data loss. Without WAL, each fsync flushes the memtable and produces a small SST, leading to compaction churn. With WAL, fsyncs write to the WAL instead, so the memtable keeps growing and flushes produce properly-sized SSTs. Enable WAL for fsync-heavy workloads to reduce compaction overhead.

• sync_writes (default: false): When enabled, every write is durably flushed before returning success, eliminating the "in-memory unflushed" window between fsync calls. This does not change POSIX fsync semantics: with the default false, explicit fsync from clients is still honored and waits for durable persistence. The flag only governs writes between fsync calls. Trades per-op latency (one round-trip per concurrent batch, since concurrent writes still coalesce) for zero unflushed data on crash. Pair with wal_enabled = true, without the WAL, every write blocks on a memtable->L0 SST flush. See the Eliminating the Durability Window section for guidance on when to enable.

When to Tune LSM Parameters

Consider tuning these parameters if you're experiencing:

• High write latency: Increase max_unflushed_gb to batch more writes
• High read latency: Decrease l0_max_ssts to reduce read amplification
• CPU / Network underutilization: Increase max_concurrent_compactions on high-core systems
• Memory pressure: Decrease max_unflushed_gb to reduce memory usage

Separate WAL Object Store

By default the WAL is written to the same object store as everything else. You can point it at a separate, lower-latency store to reduce fsync latency. See Separate WAL Store for details.

[wal]
url = "file:///mnt/nvme/zerofs-wal"
# storage_class = "..."  # Optional, independent from [storage]

The [wal] section supports its own [wal.aws], [wal.azure], and [wal.gcp] credential blocks, independent from the main storage credentials. It also has its own optional storage_class (see Storage Class); it does not inherit the one from [storage]. Since the WAL is hot, frequently-rewritten data, it is normally left on the default tier.

Multiple Instances

One read-write instance and any number of read-only instances can run on the same storage backend, all using the same configuration file. See Read Replicas for startup, freshness, and restrictions.

Complete Examples

Basic S3 Configuration

# /etc/zerofs/zerofs.toml
[cache]
dir = "/var/cache/zerofs"
disk_size_gb = 10.0

[storage]
url = "s3://my-bucket/zerofs-data"
encryption_password = "your-secure-password"

[aws]
access_key_id = "AKIAIOSFODNN7EXAMPLE"
secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
# conditional_put = "redis://localhost:6379"  # Only needed for S3-compatible stores without conditional put support

[servers.nfs]
addresses = ["0.0.0.0:2049"]

Basic GCS Configuration

On GCP VM/GKE (recommended):

# /etc/zerofs/zerofs.toml - Minimal config for GCP VMs
[cache]
dir = "/var/cache/zerofs"
disk_size_gb = 10.0

[storage]
url = "gs://my-bucket/zerofs-data"
encryption_password = "your-secure-password"

[servers.nfs]
addresses = ["0.0.0.0:2049"]

# No [gcp] section needed - uses VM's attached service account automatically

With Service Account Key File:

# /etc/zerofs/zerofs.toml
[cache]
dir = "/var/cache/zerofs"
disk_size_gb = 10.0

[storage]
url = "gs://my-bucket/zerofs-data"
encryption_password = "your-secure-password"

[gcp]
service_account = "/path/to/service-account-key.json"

[servers.nfs]
addresses = ["0.0.0.0:2049"]

Or using GOOGLE_APPLICATION_CREDENTIALS:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
zerofs run -c zerofs.toml

Performance-Tuned Configuration

# /etc/zerofs/zerofs-performance.toml
[cache]
dir = "/nvme/zerofs-cache"    # Use fast NVMe storage
disk_size_gb = 100.0
memory_size_gb = 16.0          # Large memory cache

[storage]
url = "s3://my-bucket/zerofs-data"
encryption_password = "very-secure-password-here"

[filesystem]
max_size_gb = 500.0            # Optional: limit filesystem size
compression = "zstd-3"         # Optional: "zstd-{1-22}" (default: "zstd-3") or "lz4"

[lsm]
l0_max_ssts = 128              
max_unflushed_gb = 2.0        
max_concurrent_compactions = 16  
flush_interval_secs = 60      

[aws]
access_key_id = "your-key"
secret_access_key = "your-secret"
region = "us-east-1"

[servers.nfs]
addresses = ["0.0.0.0:2049"]

[servers.nbd]
addresses = ["0.0.0.0:10809"]
unix_socket = "/tmp/zerofs.nbd.sock"  # Unix socket for local clients

S3-Compatible Services

[cache]
dir = "/var/cache/zerofs"
disk_size_gb = 10.0

[storage]
url = "s3://my-bucket/path"
encryption_password = "secure-password"

[aws]
access_key_id = "minioadmin"
secret_access_key = "minioadmin"
endpoint = "https://minio.example.com"
allow_http = "true"  # Quoted string "true", not a bare boolean

[servers.nfs]
addresses = ["0.0.0.0:2049"]

Running ZeroFS

With Configuration File

# Start ZeroFS with a config file
zerofs run --config /etc/zerofs/zerofs.toml

# Or use the shorthand
zerofs run -c zerofs.toml

Password Management

To change the encryption password:

# Change password interactively
zerofs change-password --config zerofs.toml

# The command will:
# 1. Prompt for the new password
# 2. Update encrypted data with the new password
# 3. You'll need to update the config file manually

Environment Variable Substitution

Configuration values can reference environment variables using $VAR or ${VAR} syntax:

[storage]
url = "s3://my-bucket/data"
encryption_password = "${ZEROFS_PASSWORD}"

[aws]
access_key_id = "${AWS_ACCESS_KEY_ID}"
secret_access_key = "${AWS_SECRET_ACCESS_KEY}"

Substitution applies to these keys only; all other values are taken literally:

• url and encryption_password under [storage]
• dir under [cache]
• url under [wal]
• unix_socket under [servers.ninep], [servers.nbd], and [servers.rpc]
• every value in [aws], [azure], and [gcp], including their [wal.*] variants

If a referenced variable is unset, the configuration fails to load with Failed to expand environment variable. A literal $ in an expandable value — an encryption password, for example — must be written $$.

This is useful for:

• Keeping secrets out of configuration files
• Using the same config across environments
• Integration with secret management systems

System Integration

systemd Service

Create /etc/systemd/system/zerofs.service:

[Unit]
Description=ZeroFS S3 Filesystem
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/zerofs run --config /etc/zerofs/zerofs.toml
Restart=always
RestartSec=5
# Optional: Load environment variables for substitution
EnvironmentFile=-/etc/zerofs/zerofs.env

[Install]
WantedBy=multi-user.target

If using environment variable substitution, create /etc/zerofs/zerofs.env:

ZEROFS_PASSWORD=your-secure-password
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret

Docker

# Create config file
cat > zerofs.toml <<EOF
[cache]
dir = "/cache"
disk_size_gb = 10.0

[storage]
url = "s3://bucket/path"
encryption_password = "\${ZEROFS_PASSWORD}"

[aws]
access_key_id = "\${AWS_ACCESS_KEY_ID}"
secret_access_key = "\${AWS_SECRET_ACCESS_KEY}"

[servers.nfs]
addresses = ["0.0.0.0:2049"]
EOF

# Run container
docker run -d \
  -e ZEROFS_PASSWORD='secure-password' \
  -e AWS_ACCESS_KEY_ID='your-key' \
  -e AWS_SECRET_ACCESS_KEY='your-secret' \
  -v $(pwd)/zerofs.toml:/config/zerofs.toml:ro \
  -v /tmp/cache:/cache \
  -p 2049:2049 \
  ghcr.io/barre/zerofs:latest run --config /config/zerofs.toml

Configuration Best Practices

1. Restrict permissions: chmod 600 /etc/zerofs/zerofs.toml
2. Use environment variables for secrets: Keep passwords out of config files
3. Version control: Track config files (without secrets) in git
4. Use strong passwords: Generate with openssl rand -base64 32
5. Separate environments: Use different config files for dev/staging/prod

Logging Configuration

Set log levels using the RUST_LOG environment variable. With RUST_LOG unset, zerofs run logs at info for all crates; zerofs mount uses info,fuser::reply=off. Logs are written to stderr.

# Override the default filter
export RUST_LOG='zerofs=debug'

# Common log levels:
# - error: Only errors
# - warn: Warnings and errors
# - info: Informational messages (default)
# - debug: Detailed debugging
# - trace: Very detailed tracing

# Run with custom logging
RUST_LOG=zerofs=debug zerofs run -c zerofs.toml

Next Steps