Troubleshooting
This guide helps you diagnose and resolve common issues with ZeroFS. From mount failures to performance problems, we'll walk through systematic debugging steps and solutions for the most frequent problems users encounter.
Enable debug logging with export RUST_LOG=zerofs=debug to get detailed information about what ZeroFS is doing internally.
Common Issues
Mount Failures
# Error: mount.nfs: access denied by server
# Solution: Check NFS port and permissions
# Verify ZeroFS is running
ps aux | grep zerofs
# Check if port is listening
netstat -tlnp | grep 2049
# or
ss -tlnp | grep 2049
# Try mounting with verbose output
sudo mount -v -t nfs -o vers=3,tcp,port=2049 127.0.0.1:/ /mnt/zerofs
S3 Connectivity Issues
# Error: InvalidAccessKeyId or SignatureDoesNotMatch
# Solution: Verify AWS credentials in config
# Test credentials directly
aws s3 ls s3://your-bucket/ --debug
# Check configuration file
cat zerofs.toml | grep -A 5 '\[aws\]'
# Verify values are correct:
# [aws]
# access_key_id = "your-key"
# secret_access_key = "your-secret"
# region = "us-east-1"
# endpoint = "https://s3.wasabisys.com" # For S3-compatible
Cache Problems
# Error: Failed to create cache directory
# Solution: Check permissions and disk space
# Verify cache directory from config
grep -A 3 '\[cache\]' zerofs.toml
# Check if directory exists
ls -la /var/cache/zerofs
# Check disk space
df -h /var/cache/zerofs
# Fix permissions
sudo mkdir -p /var/cache/zerofs
sudo chown $USER:$USER /var/cache/zerofs
chmod 755 /var/cache/zerofs
# Verify write access
touch /var/cache/zerofs/test && rm /var/cache/zerofs/test
Configuration Issues
Missing Configuration File
# Error: Failed to load config
# Solution: Create or specify config file
# Initialize a new config
zerofs init
# Run with specific config file
zerofs run --config /path/to/zerofs.toml
# Or use short form
zerofs run -c zerofs.toml
Encryption Issues
Password Problems
# Wrong password or corrupted data
# Verify password in config file:
[storage]
url = "s3://bucket/path"
encryption_password = "correct-password"
# For special characters, use single quotes in TOML:
encryption_password = 'p@$$w0rd!'
# Or use environment variable substitution:
encryption_password = "${ZEROFS_PASSWORD}"
Changing Passwords
# Change password using the CLI
zerofs change-password --config zerofs.toml
# You'll be prompted for the new password
# After changing, update your config file:
# [storage]
# encryption_password = "new-password"
NBD Issues
Device Connection
# Error: nbd-client connection failed
# Solution: Check NBD configuration and availability
# Verify NBD is configured in zerofs.toml:
# [servers.nbd]
# addresses = ["127.0.0.1:10809"]
# Check if port is listening
ss -tlnp | grep 10809
# Ensure NBD module is loaded (Linux)
sudo modprobe nbd
lsmod | grep nbd
# Create NBD device file
sudo mkdir -p /mnt/zerofs/.nbd
sudo truncate -s 10G /mnt/zerofs/.nbd/my-device
# Connect to the device
sudo nbd-client 127.0.0.1 10809 /dev/nbd0 -N my-device
Debugging Techniques
Enable Detailed Logging
# Maximum verbosity
export RUST_LOG='zerofs=trace,slatedb=debug'
# Log to file for analysis
export RUST_LOG='zerofs=debug'
zerofs run -c zerofs.toml 2>&1 | tee zerofs.log
# Filter specific components
export RUST_LOG='zerofs::nfs=trace,zerofs::cache=debug'