Troubleshooting
This guide helps you diagnose and resolve common issues with ZeroFS. From mount failures to performance problems, we'll walk through systematic debugging steps and solutions for the most frequent problems users encounter.
Enable debug logging with export RUST_LOG=zerofs=debug to get detailed information about what ZeroFS is doing internally.
Common Issues
Mount Failures
# Error: mount.nfs: access denied by server
# Solution: Check NFS port and permissions
# Verify ZeroFS is running
ps aux | grep zerofs
# Check if port is listening
netstat -tlnp | grep 2049
# or
ss -tlnp | grep 2049
# Try mounting with verbose output
sudo mount -v -t nfs -o vers=3,tcp,port=2049 127.0.0.1:/ /mnt/zerofs
S3 Connectivity Issues
# Error: InvalidAccessKeyId or SignatureDoesNotMatch
# Solution: Verify AWS credentials
# Test credentials directly
aws s3 ls s3://your-bucket/ --debug
# Check environment variables
echo $AWS_ACCESS_KEY_ID
echo $AWS_SECRET_ACCESS_KEY
echo $AWS_DEFAULT_REGION
# For S3-compatible services
export AWS_ENDPOINT='https://s3.wasabisys.com'
export AWS_ALLOW_HTTP='false'
Cache Problems
# Error: Failed to create cache directory
# Solution: Check permissions and disk space
# Verify cache directory exists
ls -la $SLATEDB_CACHE_DIR
# Check disk space
df -h $SLATEDB_CACHE_DIR
# Fix permissions
sudo mkdir -p /var/cache/zerofs
sudo chown $USER:$USER /var/cache/zerofs
chmod 755 /var/cache/zerofs
# Verify write access
touch $SLATEDB_CACHE_DIR/test && rm $SLATEDB_CACHE_DIR/test
Encryption Issues
Password Problems
# Error: Failed to decrypt
# Wrong password or corrupted data
# Verify password is correct
export ZEROFS_ENCRYPTION_PASSWORD='correct-password'
# Check for special characters that need escaping
# Use single quotes to avoid shell interpretation
export ZEROFS_ENCRYPTION_PASSWORD='p@$$w0rd!'
# If password was changed, use new password
export ZEROFS_ENCRYPTION_PASSWORD='new-password'
Changing Passwords
# Error: Password change failed
# Solution: Ensure both passwords are set correctly
export ZEROFS_ENCRYPTION_PASSWORD='current-password'
export ZEROFS_NEW_PASSWORD='new-password'
zerofs s3://bucket/path
# Verify change worked by running with new password
unset ZEROFS_NEW_PASSWORD
export ZEROFS_ENCRYPTION_PASSWORD='new-password'
zerofs s3://bucket/path
NBD Issues
Device Connection
# Error: nbd-client connection failed
# Solution: Check ports and device availability
# Verify NBD ports are configured
echo $ZEROFS_NBD_PORTS
echo $ZEROFS_NBD_DEVICE_SIZES_GB
# Check if ports are listening
ss -tlnp | grep 10809
# Ensure NBD module is loaded (Linux)
sudo modprobe nbd
lsmod | grep nbd
# Connect with explicit device name
nbd-client -N device_10809 127.0.0.1 10809 /dev/nbd0
Device Size Mismatches
# Error: Device size cannot be changed
# Solution: Delete and recreate device
# Remove existing device file
rm /mnt/zerofs/.nbd/device_10809
# Restart ZeroFS with new size
export ZEROFS_NBD_DEVICE_SIZES_GB='100'
zerofs s3://bucket/path
Debugging Techniques
Enable Detailed Logging
# Maximum verbosity
export RUST_LOG='zerofs=trace,slatedb=debug'
# Log to file for analysis
export RUST_LOG='zerofs=debug'
zerofs s3://bucket/path 2>&1 | tee zerofs.log
# Filter specific components
export RUST_LOG='zerofs::nfs=trace,zerofs::cache=debug'
System Monitoring
# Monitor CPU and memory
htop
# or
top -p $(pgrep zerofs)
# I/O statistics
iotop -p $(pgrep zerofs)
# Network traffic
iftop -i eth0 # Watch for S3 traffic
Recovery Procedures
Corrupted Cache
# Symptoms: Crashes, data inconsistency
# Solution: Clear and rebuild cache
# Stop ZeroFS
systemctl stop zerofs
# Backup cache (optional)
mv $SLATEDB_CACHE_DIR $SLATEDB_CACHE_DIR.backup
# Clear cache
rm -rf $SLATEDB_CACHE_DIR/*
# Restart ZeroFS
systemctl start zerofs
# Cache will rebuild automatically