# S3 Backup & Disaster Recovery for SQLite Queues Automate S3 backups for bunqueue's SQLite database. Backup scheduling, retention policies, and step-by-step disaster recovery guide. Canonical: https://bunqueue.dev/blog/s3-backup-recovery/ --- import { Aside, Steps } from '@astrojs/starlight/components';
blog ยท operations

Disaster recovery, one S3 backup.

bunqueue stores everything in a single SQLite file, which makes backups simple, but simple does not mean optional. Set up automated S3 backups, retention, and a tested recovery path.

## Why Backup? SQLite is crash-safe (WAL mode + fsync), but it can't protect against: - **Disk failure** - hardware dies, data is gone - **Accidental deletion** - `rm -rf` happens - **Corruption** - filesystem bugs, power loss during write - **Migration errors** - bad deploy wipes the data directory S3 backup gives you point-in-time recovery with minimal effort. ## Enabling S3 Backup Configure via environment variables or a [configuration file](/guide/configuration/): ```bash # Required BUNQUEUE_DATA_PATH=/var/lib/bunqueue/queue.db S3_BACKUP_ENABLED=1 S3_BUCKET=my-bunqueue-backups S3_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE S3_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY # Optional S3_REGION=us-east-1 # Default: us-east-1 S3_ENDPOINT= # Custom endpoint (MinIO, R2, etc.) S3_SESSION_TOKEN= # Temporary credentials, when used S3_VIRTUAL_HOSTED_STYLE=false # Set true only when provider requires it S3_BACKUP_INTERVAL=21600000 # Every 6 hours (default) S3_BACKUP_RETENTION=7 # Keep the last 7 backups (default) S3_BACKUP_PREFIX=backups/ # Key prefix (default) ``` Or pass them when starting the server: ```bash S3_BACKUP_ENABLED=1 \ S3_BUCKET=my-backups \ S3_REGION=us-east-1 \ bunqueue start --data-path ./data/queue.db ``` ## Compatible Storage Providers Any S3-compatible storage works: | Provider | S3_ENDPOINT | Notes | |----------|-------------|-------| | AWS S3 | *(empty)* | Default | | Cloudflare R2 | `https://.r2.cloudflarestorage.com` | No egress fees | | MinIO | `http://minio:9000` | Self-hosted | | DigitalOcean Spaces | `https://.digitaloceanspaces.com` | Simple setup | | Backblaze B2 | `https://s3..backblazeb2.com` | Cheapest storage | ```bash # Cloudflare R2 example S3_BACKUP_ENABLED=1 S3_BUCKET=bunqueue-backups S3_ENDPOINT=https://abc123.r2.cloudflarestorage.com S3_ACCESS_KEY_ID=your-r2-key S3_SECRET_ACCESS_KEY=your-r2-secret S3_REGION=auto ``` ## Backup Process The backup runs on a timer (default: every 6 hours): 1. **Flush and snapshot** - drains pending buffered writes, then SQLite `VACUUM INTO` creates a standalone snapshot that includes committed WAL frames 2. **Validate and compress** - runs `PRAGMA integrity_check`, computes SHA-256 over the original bytes, then gzip-compresses 3. **Publish to S3** - uploads metadata first and the uniquely named payload second as the commit point 4. **Cleanup old backups** - keeps only the most recent `S3_BACKUP_RETENTION` payload/metadata pairs ## Backup File Naming Backups are stored under the `S3_BACKUP_PREFIX` (default `backups/`) with this key pattern: ``` backups/ bunqueue-2026-01-15T00-00-00-000Z-.db (gzip-compressed) bunqueue-2026-01-15T00-00-00-000Z-.db.meta.json (checksum + sizes) ``` Note that the `.db` object is gzip-compressed even though the key has no `.gz` suffix; the `.meta.json` sidecar records the compression flag and a SHA-256 checksum of the original database. ## Disaster Recovery The built-in restore command handles download, decompression, and validation for you: 1. **Stop bunqueue** ```bash systemctl stop bunqueue # or: docker stop bunqueue ``` 2. **Find the backup to restore** ```bash # Requires the same S3_* env vars plus BUNQUEUE_DATA_PATH bunqueue backup list ``` 3. **Restore it** ```bash bunqueue backup restore 'backups/bunqueue-2026-01-15T18-00-00-000Z-.db' --force ``` The restore is validate-before-replace: it verifies metadata, compressed/original sizes and SHA-256, checks the SQLite header, runs an integrity check on a temp file, quarantines stale WAL/SHM/journal sidecars, and only then atomically swaps it into place. On a pre-swap failure the live database is left untouched. 4. **Restart bunqueue** ```bash systemctl start bunqueue # or: docker start bunqueue ``` Prefer the CLI because it authenticates and validates the candidate. If an emergency requires a manual restore, remember that the object is gzip-compressed despite the `.db` key, verify the metadata checksum yourself, and remove stale sidecars while the server is stopped: ```bash aws s3 cp 's3://my-bunqueue-backups/backups/.db' ./backup.db.gz aws s3 cp 's3://my-bunqueue-backups/backups/.db.meta.json' ./backup.meta.json gzip -dc backup.db.gz > /var/lib/bunqueue/queue.db.candidate sqlite3 /var/lib/bunqueue/queue.db.candidate 'PRAGMA integrity_check' mv /var/lib/bunqueue/queue.db /var/lib/bunqueue/queue.db.corrupted rm -f /var/lib/bunqueue/queue.db-wal /var/lib/bunqueue/queue.db-shm /var/lib/bunqueue/queue.db-journal mv /var/lib/bunqueue/queue.db.candidate /var/lib/bunqueue/queue.db ``` bunqueue will recover the queue state from the restored database, reloading pending jobs, cron schedules, and DLQ entries. ## Backup Monitoring Monitor backup health in production: ```bash # Check the backup configuration bunqueue backup status # List backups and check the most recent timestamp bunqueue backup list # Trigger a backup on demand bunqueue backup now ``` Set up alerts for: - **No backup in 2x the interval** - backup process may be failing - **Backup size anomalies** - sudden size changes may indicate issues - **S3 upload failures** - check credentials and bucket permissions ## Supplementary Strategies S3 backup covers most scenarios, but consider layering additional protection: **Filesystem Snapshots** ```bash # LVM snapshot (instant, zero downtime) lvcreate -s -n bunqueue-snap -L 1G /dev/vg0/bunqueue # ZFS snapshot zfs snapshot tank/bunqueue@daily ``` **Cron-based local backup** ```bash # Copy SQLite file every hour (WAL must be checkpointed first) 0 * * * * sqlite3 /var/lib/bunqueue/queue.db ".backup /backups/queue-$(date +\%H).db" ``` **Replication to secondary server** ```bash # rsync the database file periodically */30 * * * * rsync -az /var/lib/bunqueue/ backup-server:/bunqueue-replica/ ``` ## Best Practices 1. **Enable S3 backup from day one** - don't wait for your first data loss 2. **Test recovery regularly** - a backup you can't restore from is worthless 3. **Monitor backup health** - alert on missed backups 4. **Use retention policies** - keep the last 7-30 backups depending on your needs 5. **Consider R2 or B2** for cost-effective storage (no egress fees with R2)