blog · operations

Disaster recovery, one S3 backup.

bunqueue stores everything in a single SQLite file, which makes backups simple, but simple does not mean optional. Set up automated S3 backups, retention, and a tested recovery path.

Why Backup?

SQLite is crash-safe (WAL mode + fsync), but it can’t protect against:

• Disk failure - hardware dies, data is gone
• Accidental deletion - rm -rf happens
• Corruption - filesystem bugs, power loss during write
• Migration errors - bad deploy wipes the data directory

S3 backup gives you point-in-time recovery with minimal effort.

Enabling S3 Backup

Configure via environment variables or a configuration file:

# Required
BUNQUEUE_DATA_PATH=/var/lib/bunqueue/queue.db
S3_BACKUP_ENABLED=1
S3_BUCKET=my-bunqueue-backups
S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

# Optional
S3_REGION=us-east-1               # Default: us-east-1
S3_ENDPOINT=                      # Custom endpoint (MinIO, R2, etc.)
S3_SESSION_TOKEN=                 # Temporary credentials, when used
S3_VIRTUAL_HOSTED_STYLE=false     # Set true only when provider requires it
S3_BACKUP_INTERVAL=21600000       # Every 6 hours (default)
S3_BACKUP_RETENTION=7             # Keep the last 7 backups (default)
S3_BACKUP_PREFIX=backups/         # Key prefix (default)

Or pass them when starting the server:

S3_BACKUP_ENABLED=1 \
S3_BUCKET=my-backups \
S3_REGION=us-east-1 \
bunqueue start --data-path ./data/queue.db

Compatible Storage Providers

Any S3-compatible storage works:

Provider	S3_ENDPOINT	Notes
AWS S3	(empty)	Default
Cloudflare R2	https://<account>.r2.cloudflarestorage.com	No egress fees
MinIO	http://minio:9000	Self-hosted
DigitalOcean Spaces	https://<region>.digitaloceanspaces.com	Simple setup
Backblaze B2	https://s3.<region>.backblazeb2.com	Cheapest storage

# Cloudflare R2 example
S3_BACKUP_ENABLED=1
S3_BUCKET=bunqueue-backups
S3_ENDPOINT=https://abc123.r2.cloudflarestorage.com
S3_ACCESS_KEY_ID=your-r2-key
S3_SECRET_ACCESS_KEY=your-r2-secret
S3_REGION=auto

Backup Process

The backup runs on a timer (default: every 6 hours):

1. Flush and snapshot - drains pending buffered writes, then SQLite VACUUM INTO creates a standalone snapshot that includes committed WAL frames
2. Validate and compress - runs PRAGMA integrity_check, computes SHA-256 over the original bytes, then gzip-compresses
3. Publish to S3 - uploads metadata first and the uniquely named payload second as the commit point
4. Cleanup old backups - keeps only the most recent S3_BACKUP_RETENTION payload/metadata pairs

Note

The backup process is non-blocking. It creates a snapshot without stopping the queue, so jobs continue processing during backup.

Backup File Naming

Backups are stored under the S3_BACKUP_PREFIX (default backups/) with this key pattern:

backups/
  bunqueue-2026-01-15T00-00-00-000Z-<uuid>.db            (gzip-compressed)
  bunqueue-2026-01-15T00-00-00-000Z-<uuid>.db.meta.json  (checksum + sizes)

Note that the .db object is gzip-compressed even though the key has no .gz suffix; the .meta.json sidecar records the compression flag and a SHA-256 checksum of the original database.

Disaster Recovery

The built-in restore command handles download, decompression, and validation for you:

1. Stop bunqueue

   systemctl stop bunqueue
   # or: docker stop bunqueue

2. Find the backup to restore

   # Requires the same S3_* env vars plus BUNQUEUE_DATA_PATH
   bunqueue backup list

3. Restore it

   bunqueue backup restore 'backups/bunqueue-2026-01-15T18-00-00-000Z-<uuid>.db' --force

   The restore is validate-before-replace: it verifies metadata, compressed/original sizes and SHA-256, checks the SQLite header, runs an integrity check on a temp file, quarantines stale WAL/SHM/journal sidecars, and only then atomically swaps it into place. On a pre-swap failure the live database is left untouched.

4. Restart bunqueue

   systemctl start bunqueue
   # or: docker start bunqueue

Prefer the CLI because it authenticates and validates the candidate. If an emergency requires a manual restore, remember that the object is gzip-compressed despite the .db key, verify the metadata checksum yourself, and remove stale sidecars while the server is stopped:

aws s3 cp 's3://my-bunqueue-backups/backups/<exact-key>.db' ./backup.db.gz
aws s3 cp 's3://my-bunqueue-backups/backups/<exact-key>.db.meta.json' ./backup.meta.json
gzip -dc backup.db.gz > /var/lib/bunqueue/queue.db.candidate
sqlite3 /var/lib/bunqueue/queue.db.candidate 'PRAGMA integrity_check'
mv /var/lib/bunqueue/queue.db /var/lib/bunqueue/queue.db.corrupted
rm -f /var/lib/bunqueue/queue.db-wal /var/lib/bunqueue/queue.db-shm /var/lib/bunqueue/queue.db-journal
mv /var/lib/bunqueue/queue.db.candidate /var/lib/bunqueue/queue.db

bunqueue will recover the queue state from the restored database, reloading pending jobs, cron schedules, and DLQ entries.

Caution

Recovery restores to the backup point-in-time. Jobs added or completed between the backup and the failure are lost. Reduce S3_BACKUP_INTERVAL for critical workloads.

Backup Monitoring

Monitor backup health in production:

# Check the backup configuration
bunqueue backup status

# List backups and check the most recent timestamp
bunqueue backup list

# Trigger a backup on demand
bunqueue backup now

Set up alerts for:

• No backup in 2x the interval - backup process may be failing
• Backup size anomalies - sudden size changes may indicate issues
• S3 upload failures - check credentials and bucket permissions

Supplementary Strategies

S3 backup covers most scenarios, but consider layering additional protection:

Filesystem Snapshots

# LVM snapshot (instant, zero downtime)
lvcreate -s -n bunqueue-snap -L 1G /dev/vg0/bunqueue

# ZFS snapshot
zfs snapshot tank/bunqueue@daily

Cron-based local backup

# Copy SQLite file every hour (WAL must be checkpointed first)
0 * * * * sqlite3 /var/lib/bunqueue/queue.db ".backup /backups/queue-$(date +\%H).db"

Replication to secondary server

# rsync the database file periodically
*/30 * * * * rsync -az /var/lib/bunqueue/ backup-server:/bunqueue-replica/

Best Practices

1. Enable S3 backup from day one - don’t wait for your first data loss
2. Test recovery regularly - a backup you can’t restore from is worthless
3. Monitor backup health - alert on missed backups
4. Use retention policies - keep the last 7-30 backups depending on your needs
5. Consider R2 or B2 for cost-effective storage (no egress fees with R2)