Configuration Guide

ZeroFS uses TOML configuration files for all settings. This guide covers all available configuration options to help you configure ZeroFS for your specific use case.

Getting Started

Create a configuration file using the init command:

zerofs init

This creates a template configuration file that you can customize.

Configuration File Structure

Basic Configuration

# Cache configuration (required)
[cache]
dir = "/var/cache/zerofs"        # Directory for caching data
disk_size_gb = 10.0              # Maximum disk cache size in GB
memory_size_gb = 2.0             # Memory cache size in GB (optional)

# Storage configuration (required)
[storage]
url = "s3://bucket/path"         # Storage backend URL
encryption_password = "your-secure-password"  # Encryption password

Storage Backends

AWS S3

[storage]
url = "s3://my-bucket/zerofs-data"
encryption_password = "secure-password-here"

[aws]
access_key_id = "AKIAIOSFODNN7EXAMPLE"
secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
region = "us-east-1"              # Optional, defaults to us-east-1
endpoint = "https://s3.amazonaws.com"  # Optional for S3-compatible services
allow_http = "false"                # Set to true for non-HTTPS endpoints

For S3-compatible services like MinIO or Cloudflare R2, set the endpoint field to the service endpoint.

Azure Blob Storage

[storage]
url = "azure://container/path"
encryption_password = "secure-password-here"

[azure]
storage_account_name = "myaccount"
storage_account_key = "your-account-key"

Google Cloud Storage (GCS)

Using Application Default Credentials (GCP VMs/GKE):

[storage]
url = "gs://my-bucket/zerofs-data"
encryption_password = "secure-password-here"

# No [gcp] section needed when running on GCP with attached service account

Using Service Account Key File:

[storage]
url = "gs://my-bucket/zerofs-data"
encryption_password = "secure-password-here"

[gcp]
service_account = "/path/to/service-account-key.json"
# Or use application_credentials = "${GOOGLE_APPLICATION_CREDENTIALS}"

GCS authentication follows Google's Application Default Credentials (ADC) chain:

If running on a GCP VM or GKE pod with an attached service account, credentials are automatically discovered via the metadata service
Otherwise, uses GOOGLE_APPLICATION_CREDENTIALS environment variable or [gcp] config section
Falls back to gcloud CLI credentials if available

Local Filesystem

[storage]
url = "file:///path/to/storage"
encryption_password = "secure-password-here"

# No additional backend configuration needed

Network Services

NFS Server

[servers.nfs]
addresses = ["0.0.0.0:2049"]  # Bind addresses (default: ["127.0.0.1:2049"])

9P Server

[servers.ninep]
addresses = ["0.0.0.0:5564"]  # Bind addresses (default: ["127.0.0.1:5564"])
unix_socket = "/tmp/zerofs.sock"

NBD Server

[servers.nbd]
addresses = ["0.0.0.0:10809"]  # Bind addresses (default: ["127.0.0.1:10809"])
unix_socket = "/tmp/zerofs-nbd.sock"

Filesystem Quotas

ZeroFS supports configurable filesystem size limits:

[filesystem]
max_size_gb = 100.0  # Limit filesystem to 100 GB

When the quota is reached, write operations return ENOSPC (No space left on device). Delete and truncate operations continue to work, allowing you to free space. If not specified, the filesystem defaults to 16 EiB (effectively unlimited).

Compression

ZeroFS compresses file data (chunks) before encryption. You can choose between two compression algorithms:

[filesystem]
compression = "lz4"      # Fast compression (default)
# or
compression = "zstd-3"   # Zstd with level 3

Algorithms

lz4 (default): Very fast compression and decompression with moderate compression ratio. Best for workloads where speed is critical.
zstd-{level}: Zstandard compression with configurable level from 1 to 22. Lower levels (1-5) are faster, higher levels (15-22) achieve better compression but are slower.

Changing Compression On-the-fly

You can change the compression algorithm at any time without migration:

New writes use the currently configured algorithm
Existing data remains readable regardless of how it was compressed
Compression format is auto-detected per-chunk when reading

LSM Tree Performance Tuning

ZeroFS uses an LSM (Log-Structured Merge) tree as its underlying storage engine. For advanced performance tuning, you can configure LSM parameters:

[lsm]
l0_max_ssts = 16                 # Max SST files in L0 before compaction
max_unflushed_gb = 1.0           # Max unflushed data before forcing flush (in GB)
max_concurrent_compactions = 8   # Max concurrent compaction operations
flush_interval_secs = 30         # Interval between periodic flushes (in seconds)

Parameters

l0_max_ssts (default: 16, min: 4): Maximum number of SST (Sorted String Table) files allowed in level 0 before triggering compaction. Lower values reduce read amplification but increase write amplification. Higher values do the opposite.
max_unflushed_gb (default: 1.0, min: 0.1): Maximum amount of unflushed data (in gigabytes) before forcing a flush to storage. Supports fractional values like 0.5 for 500 MB. Larger values improve write batching but increase memory usage.
max_concurrent_compactions (default: 8, min: 1): Maximum number of compaction operations that can run concurrently. Higher values can improve throughput on systems with many CPU cores and high network throughput but increase memory usage.
flush_interval_secs (default: 30, min: 5): Interval in seconds between periodic background flushes to durable storage. Lower values flush more frequently. Higher values improve write throughput by batching more data before flushing. Note that fsync calls on NBD and 9P always trigger an immediate flush regardless of this interval.

These are advanced settings. The defaults are optimized for most workloads. Only modify these if you understand LSM tree behavior and are experiencing specific performance issues.

When to Tune LSM Parameters

Consider tuning these parameters if you're experiencing:

High write latency: Increase max_unflushed_gb to batch more writes
High read latency: Decrease l0_max_ssts to reduce read amplification
CPU / Network underutilization: Increase max_concurrent_compactions on high-core systems
Memory pressure: Decrease max_unflushed_gb to reduce memory usage

Multiple Instances

ZeroFS supports running multiple instances on the same storage backend: one read-write instance and multiple read-only instances.

# Read-write instance (default)
zerofs run -c zerofs.toml

# Read-only instances
zerofs run -c zerofs.toml --read-only

Read-only instances automatically see updates from the writer and return EROFS errors for write operations.

Only one read-write instance can run at a time. Multiple read-only instances can run safely alongside the writer for load balancing.

Complete Examples

Basic S3 Configuration

# /etc/zerofs/zerofs.toml
[cache]
dir = "/var/cache/zerofs"
disk_size_gb = 10.0

[storage]
url = "s3://my-bucket/zerofs-data"
encryption_password = "your-secure-password"

[aws]
access_key_id = "AKIAIOSFODNN7EXAMPLE"
secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"

[servers.nfs]
addresses = ["0.0.0.0:2049"]

Basic GCS Configuration

On GCP VM/GKE (recommended):

# /etc/zerofs/zerofs.toml - Minimal config for GCP VMs
[cache]
dir = "/var/cache/zerofs"
disk_size_gb = 10.0

[storage]
url = "gs://my-bucket/zerofs-data"
encryption_password = "your-secure-password"

[servers.nfs]
addresses = ["0.0.0.0:2049"]

# No [gcp] section needed - uses VM's attached service account automatically

With Service Account Key File:

# /etc/zerofs/zerofs.toml
[cache]
dir = "/var/cache/zerofs"
disk_size_gb = 10.0

[storage]
url = "gs://my-bucket/zerofs-data"
encryption_password = "your-secure-password"

[gcp]
service_account = "/path/to/service-account-key.json"

[servers.nfs]
addresses = ["0.0.0.0:2049"]

Or using GOOGLE_APPLICATION_CREDENTIALS:

export GOOGLE_APPLICATION_CREDENTIALS="/path/to/service-account-key.json"
zerofs run -c zerofs.toml

High-Performance Configuration

# /etc/zerofs/zerofs-performance.toml
[cache]
dir = "/nvme/zerofs-cache"    # Use fast NVMe storage
disk_size_gb = 100.0
memory_size_gb = 16.0          # Large memory cache

[storage]
url = "s3://high-performance-bucket/data"
encryption_password = "very-secure-password-here"

[filesystem]
max_size_gb = 500.0            # Optional: limit filesystem size
compression = "lz4"            # Optional: "lz4" (default) or "zstd-{1-22}"

[lsm]
l0_max_ssts = 16               # Trigger compaction with fewer SST files
max_unflushed_gb = 2.0         # Allow more data batching for better write throughput
max_concurrent_compactions = 16  # Use more CPU cores for compaction
flush_interval_secs = 60       # Flush less frequently for better throughput

[aws]
access_key_id = "your-key"
secret_access_key = "your-secret"
region = "us-east-1"

[servers.nfs]
addresses = ["0.0.0.0:2049"]

[servers.nbd]
addresses = ["0.0.0.0:10809"]
unix_socket = "/tmp/zerofs-nbd.sock"  # Unix socket for better local performance

S3-Compatible Services

[cache]
dir = "/var/cache/zerofs"
disk_size_gb = 10.0

[storage]
url = "s3://my-bucket/path"
encryption_password = "secure-password"

[aws]
access_key_id = "minioadmin"
secret_access_key = "minioadmin"
endpoint = "https://minio.example.com"
allow_http = "true"  # Set to true if using HTTP

Running ZeroFS

With Configuration File

# Start ZeroFS with a config file
zerofs run --config /etc/zerofs/zerofs.toml

# Or use the shorthand
zerofs run -c zerofs.toml

Password Management

To change the encryption password:

# Change password interactively
zerofs change-password --config zerofs.toml

# The command will:
# 1. Prompt for the new password
# 2. Update encrypted data with the new password
# 3. You'll need to update the config file manually

After changing the password, update the encryption_password field in your configuration file.

Environment Variable Substitution

Configuration values can reference environment variables using ${VAR} syntax:

[storage]
url = "s3://my-bucket/data"
encryption_password = "${ZEROFS_PASSWORD}"

[aws]
access_key_id = "${AWS_ACCESS_KEY_ID}"
secret_access_key = "${AWS_SECRET_ACCESS_KEY}"

This is useful for:

Keeping secrets out of configuration files
Using the same config across environments
Integration with secret management systems

System Integration

systemd Service

Create /etc/systemd/system/zerofs.service:

[Unit]
Description=ZeroFS S3 Filesystem
After=network-online.target
Wants=network-online.target

[Service]
Type=simple
ExecStart=/usr/local/bin/zerofs run --config /etc/zerofs/zerofs.toml
Restart=always
RestartSec=5
# Optional: Load environment variables for substitution
EnvironmentFile=-/etc/zerofs/zerofs.env

[Install]
WantedBy=multi-user.target

If using environment variable substitution, create /etc/zerofs/zerofs.env:

ZEROFS_PASSWORD=your-secure-password
AWS_ACCESS_KEY_ID=your-key
AWS_SECRET_ACCESS_KEY=your-secret

Docker

# Create config file
cat > zerofs.toml <<EOF
[cache]
dir = "/cache"
disk_size_gb = 10.0

[storage]
url = "s3://bucket/path"
encryption_password = "\${ZEROFS_PASSWORD}"

[aws]
access_key_id = "\${AWS_ACCESS_KEY_ID}"
secret_access_key = "\${AWS_SECRET_ACCESS_KEY}"

[servers.nfs]
addresses = ["0.0.0.0:2049"]
EOF

# Run container
docker run -d \
  -e ZEROFS_PASSWORD='secure-password' \
  -e AWS_ACCESS_KEY_ID='your-key' \
  -e AWS_SECRET_ACCESS_KEY='your-secret' \
  -v $(pwd)/zerofs.toml:/config/zerofs.toml:ro \
  -v /tmp/cache:/cache \
  -p 2049:2049 \
  ghcr.io/barre/zerofs:latest run --config /config/zerofs.toml

Configuration Best Practices

Restrict permissions: chmod 600 /etc/zerofs/zerofs.toml
Use environment variables for secrets: Keep passwords out of config files
Version control: Track config files (without secrets) in git
Use strong passwords: Generate with openssl rand -base64 32
Separate environments: Use different config files for dev/staging/prod

Logging Configuration

Set log levels using the RUST_LOG environment variable:

# Set log level (default: error,zerofs=info)
export RUST_LOG='zerofs=debug'

# Common log levels:
# - error: Only errors
# - warn: Warnings and errors
# - info: Informational messages (default for zerofs)
# - debug: Detailed debugging
# - trace: Very detailed tracing

# Run with custom logging
RUST_LOG=zerofs=debug zerofs run -c zerofs.toml

Next Steps

Get started with ZeroFS

Troubleshooting guide