guide · production

What survives a crash.

Running bunqueue in production is one process plus one SQLite file. This page answers, in plain words, what happens when something dies mid-job, what to monitor, and what to check before go-live.

quick answers

If it dies, what happens?

The short version first. Each row is a real failure and the behavior you get out of the box.

Failure	What happens to the job
Worker crashes mid-job	The job is redelivered to another worker after its lease expires. It may run twice, never zero times.
Server killed hard (kill -9)	Jobs written to disk survive and recover on restart. Jobs accepted in the last ~10 ms may be lost, unless they were added with durable: true.
Server restarted gracefully (SIGTERM, deploys)	Nothing is lost, even buffered jobs. Shutdown flushes all pending writes before exiting.
Job fails repeatedly	After its retry attempts are exhausted it moves to the DLQ (dead letter queue, a parking lot for failed jobs you can inspect and retry). See DLQ.
Disk fills up	The server stays up and reports itself degraded on /health. Durable pushes fail explicitly instead of pretending to succeed.
The whole machine is lost	You restore the last S3 snapshot. Your maximum data loss is the backup interval.

the crash timeline

The worker dies before the ACK.

This is the failure that defines a queue’s honesty. Here is the exact sequence.

1. A worker pulls a job. The server marks it active and gives the worker a lease: a lock with a time-to-live (default 30 seconds, the worker’s lockDuration), renewed by heartbeats while the job runs.
2. Your handler does its side effect. The email is sent, the charge is captured.
3. The worker dies before its ACK reaches the server. An ACK is the worker’s “done” confirmation; without it the server has no idea the work happened.
4. The lease expires and the server requeues the job with its attempt count incremented.
5. Another worker pulls it. Your side effect runs again.

The guarantee is at-least-once: a job is never lost, but it can run more than once. No queue on any backend can promise better, because your side effect and the acknowledgment are two separate systems and a crash can always land between them. What bunqueue does guarantee around the edges:

• A late ACK after ordinary lease expiry is accepted only while that exact generation still owns the processing slot and the job was not re-leased. If the job’s processing timeout already finalized that generation, the ACK/FAIL is instead an explicit idempotent no-op. Two generations can never both complete the same job.
• Re-adding a job with the same custom jobId is a no-op, even under heavy concurrency. Producers that retry on timeout should always set one.
• Duplicate ACKs are harmless.

What you must do

• Write handlers so running the same job twice is harmless. Put an idempotency key (derived from job.id or your own jobId) on every external side effect: payments, emails, webhooks.
• Set a custom jobId wherever the producer might retry an add.
• Mark jobs you cannot afford to lose as durable: true.

durability

Two write modes, one decision.

By default bunqueue batches writes to SQLite every 10 ms for throughput. A durable write skips the batch and hits disk before the add returns.

Mode	Throughput	Loss window on a hard crash
Buffered (default)	~100k jobs/sec	up to 10 ms of accepted jobs
durable: true per job	~10k jobs/sec	none

await queue.add('send-newsletter', data);                  // buffered, fast
await queue.add('capture-payment', data, { durable: true }); // on disk before this returns

Mixing is free: mark only the jobs that are money as durable and keep the rest batched. The 10 ms window only exists for hard crashes; graceful restarts flush everything. Measured numbers per mode are in Benchmarks.

backup and restore

One file, snapshotted to S3.

The entire queue state is one SQLite database, so disaster recovery is one file in object storage. Works with AWS S3, Cloudflare R2, and MinIO.

S3_BACKUP_ENABLED=1
S3_BUCKET=my-backups
S3_ACCESS_KEY_ID=...
S3_SECRET_ACCESS_KEY=...
S3_REGION=us-east-1            # or S3_ENDPOINT for R2/MinIO
S3_BACKUP_INTERVAL=21600000    # snapshot cadence, default 6 hours
S3_BACKUP_RETENTION=7          # snapshots kept, oldest pruned

This requires a persistent data path. Scheduled server backups flush the pending write buffer (and fail the cycle if storage backoff leaves anything pending), then use SQLite VACUUM INTO, so committed WAL data is captured even when a long-lived reader prevents checkpoint truncation.

bunqueue backup now       # snapshot immediately
bunqueue backup list      # list snapshots with size and age
bunqueue backup restore <key> --force   # stop the server first
bunqueue backup status

Restore validates before it replaces: it verifies sizes and checksum, runs an integrity check on a temp copy, quarantines stale WAL/SHM sidecars, and only then swaps it in. A corrupt backup never touches your live database. Stop the server before restoring, size the interval to how much queue state you can afford to lose, and rehearse one restore before you need it. Provider setup lives in the S3 Backup guide.

monitoring

Three endpoints, no agent.

Everything is on the HTTP port (default 6790). Point Prometheus at it or just curl it.

Endpoint	What it gives you
GET /health	healthy or degraded, uptime, version, job counts per state, memory, and a storage.diskFull flag when the disk is full
GET /healthz, /live	Bare liveness probes; remain 200 while the process responds
GET /ready	Readiness; returns 503 when persistent storage is degraded
GET /prometheus	Metrics in Prometheus text format: job counts, totals, latency, bounded per-queue gauges, process/connections, storage and backup freshness
GET /metrics	The same counters as JSON

Set METRICS_AUTH=true to require an AUTH_TOKENS bearer token on /prometheus; a missing token configuration fails closed with 503. Alert on these production symptoms:

• DLQ growth: bunqueue_jobs_dlq climbing means handlers are failing terminally.
• Queue depth: bunqueue_jobs_waiting + bunqueue_jobs_prioritized growing steadily means workers cannot keep up.
• /health reporting degraded: the disk is full.
• Backup freshness: an enabled scheduler that is stopped, failing, or has no success within two intervals puts recovery objectives at risk.
• Telemetry truncation: bunqueue_queue_metrics_omitted > 0 means per-queue drill-down is incomplete; review cardinality before raising the cap.
• Memory drift: RSS should plateau once the workload does; internal caches are hard-capped.

Dashboards and webhook alerting are covered in Monitoring.

graceful shutdown

SIGTERM is a contract.

Deploys restart the server constantly. The shutdown path is what makes that boring.

On SIGTERM the server stops accepting connections, waits up to SHUTDOWN_TIMEOUT_MS (default 30,000 ms) for active jobs to finish, then flushes every pending write and closes the database cleanly. That flush is why even non-durable jobs survive deploys.

Jobs still running when the deadline hits are not lost either: they are already persisted as active, and the next startup requeues them for retry. Size SHUTDOWN_TIMEOUT_MS a little above your longest handler, and give your orchestrator a termination grace period slightly above that (Kubernetes defaults to 30 s).

go-live checklist

Before you point traffic at it.

• durable: true on every job you cannot re-derive
• idempotency keys in every handler side effect
• custom jobId wherever producers retry
• S3_BACKUP_ENABLED=1 and one rehearsed restore
• alerts on DLQ growth and waiting-queue depth
• /healthz and /ready wired into your orchestrator
• AUTH_TOKENS set; METRICS_AUTH if metrics are exposed
• native TLS or a private network between clients and server
• SHUTDOWN_TIMEOUT_MS sized above your longest handler
• disk alert covering the .db and .db-wal files
• worker lockDuration above your slowest job, or heartbeats on

Honest limits

Buffered jobs can lose up to ~10 ms of writes on a hard crash; durable: true closes that per job. bunqueue is single-instance, so losing the host means restoring the last snapshot; your recovery point is the backup interval. At-least-once means duplicates are possible by design. SQLite in WAL mode (writes append to a .db-wal sidecar file) keeps that sidecar small on its own, but watch its size on disk: sustained growth usually means a long-lived read is pinning it. When upgrading, restart the server first (the schema migrates forward automatically), then update clients.

Ready to deploy?

The deployment guide has the Docker, systemd, and PM2 configs to paste.

bun add bunqueueDeployment guideQuickstart