Storage: SQLite or PostgreSQL 15–18

guide · storage

One-file SQLite. PostgreSQL for many brokers.

Keep the zero-infrastructure SQLite engine for embedded and single-broker deployments, or opt into PostgreSQL 15–18 when several bunqueue servers must share one authoritative queue.

The storage choice is explicit and deployment-specific. SQLite behavior has not changed. PostgreSQL is a separate server backend; MySQL is not supported.

The short answer

bunqueue still needs zero external infrastructure by default. Without a data path it is memory-only; set one path for SQLite persistence. The only runtime dependency is msgpackr, and cron parsing is provided by Bun itself. If one server or embedded process and a durable disk meet your needs, use SQLite:

# Server mode: point the data path at durable storage
BUNQUEUE_DATA_PATH=/data/bunq.db bunqueue start

// Embedded mode (queue runs inside your app's process)
const queue = new Queue('jobs', { embedded: true, dataPath: '/data/jobs.db' });

Embedded mode ships in the Bun bunqueue package only. queue.forward() below is also available in bunqueue-client for Node.js and Deno. From every other language, run the server and connect with a client SDK, as in Option 3.

For multiple active broker processes, use the PostgreSQL server backend. It is database-authoritative: claims, leases, ACK/FAIL, queue limits, Pro-style group capacity/order/pause/rate state, cron schedules, worker registrations, job-state/lifecycle metrics, and events are coordinated in PostgreSQL rather than independent in-memory queues. Here, metrics means job-state and lifecycle families; process/connection and registration collectors remain broker-local.

PostgreSQL 15–18 multi-broker mode

PostgreSQL is available in standalone server mode. Embedded Queue and Worker continue to use memory/SQLite. CI runs the complete integration suite against majors 15, 16, 17, and PostgreSQL 18.6. Version 18.6 is recommended for new deployments and pinned by the repository Compose topology; the other three majors remain tested compatibility targets.

The repository Compose file pins the requested PostgreSQL version exactly:

POSTGRES_PASSWORD='replace-me' \
  BUNQUEUE_POSTGRES_URL='postgres://bunqueue:replace-me@postgres:5432/bunqueue' \
  docker compose -f docker-compose.postgres.yml up --build -d

The database password and broker URL are separate Compose inputs so a raw secret is never interpolated into a URI. When the password contains URI-reserved characters, percent-encode the password component in BUNQUEUE_POSTGRES_URL (for example, p@ss/word becomes p%40ss%2Fword) while passing the original value as POSTGRES_PASSWORD.

It starts postgres:18.6-alpine and two bunqueue brokers against the same database and namespace:

Service	TCP	HTTP
`broker-a`	`6789`	`6790`
`broker-b`	`7789`	`7790`

To configure a broker directly:

BUNQUEUE_STORAGE_DRIVER=postgres \
BUNQUEUE_POSTGRES_URL='postgres://bunqueue:secret@postgres:5432/bunqueue' \
BUNQUEUE_POSTGRES_NAMESPACE=production \
BUNQUEUE_BROKER_ID=broker-a \
bunqueue start

Or use bunqueue.config.ts:

import { defineConfig } from 'bunqueue';

export default defineConfig({
  storage: {
    driver: 'postgres',
    url: process.env.BUNQUEUE_POSTGRES_URL!,
    namespace: 'production',
    brokerId: process.env.HOSTNAME,
    poolSize: 4,
    leaseDurationMs: 30_000,
    pollIntervalMs: 250,
    statementTimeoutMs: 30_000,
    lockTimeoutMs: 5_000,
    idleTransactionTimeoutMs: 30_000,
    maxConcurrentOperations: 16,
    maxQueuedOperations: 128,
    maxSnapshotJobs: 100_000,
    maxSnapshotPayloadBytes: 256 * 1024 * 1024,
  },
});

Every active broker needs a unique brokerId and the same URL/namespace. A live duplicate fails startup. Each process also owns an internal random session fence, so a stale process cannot release leases or workers belonging to its successor. Credentials should come from a secret manager. Do not also set a SQLite data path: ambiguous PostgreSQL + SQLite configuration fails startup. The built-in S3 snapshot feature is SQLite-only; use normal PostgreSQL backup/PITR tooling for this backend.

The broker count is not limited to the two services in the example. The PostgreSQL integration gate also starts four independent bunqueue processes, each with its own ports and connection pool, against one PostgreSQL 18.6 namespace. It exercises concurrent production/consumption, shared rate and concurrency limits, group max-size admission, priority/FIFO rotation, group pause/manual deadlines, queue pause/resume, cross-broker ACK, and recovery after one broker is killed while holding leases.

PostgreSQL transactions use FOR UPDATE SKIP LOCKED for competing claims and opaque, database-clock leases for fencing. LISTEN/NOTIFY only wakes other brokers; a transactional outbox ordered at commit time repairs missed or coalesced notifications even when physical event IDs were allocated in a different order. An expired owner cannot ACK a generation after another broker recovers it. Dependency admission queries PostgreSQL and rechecks existence in the write transaction, so broker B can reference a parent just committed on broker A without relying on event timing. Worker and cron views in dashboards, per-queue worker routes, and streamed stats likewise read the shared registry.

PostgreSQL-specific bulk paths keep database work proportional to the command: one worker heartbeat batch uses one fenced transaction and one set-based update, with a pre-write generation ticket preventing late responses from replacing a newer terminal or re-leased local projection, dependency-free PUSHB validation does not materialize the local compatibility snapshot, and dashboard queue counts aggregate that snapshot once. Event retention uses exact transaction-private deltas and a consolidated per-queue count, so under-cap writes do not scan the retained journal window or lock shared counter rows. These optimizations do not change the memory or SQLite engines.

Completion proofs used by live dependencies are pinned in PostgreSQL; unused removeOnComplete proofs and each broker’s completed/result snapshot are bounded independently. Reusing a custom ID retires the previous proof only when no live consumer still owns it. Destructive commands use the same identity lock as admission, so drain, clean, TTL/DLQ pruning, and obliterate cannot strand a waiting-children job. PostgreSQL also requires at least one retained durable event per queue; its runtime rejects maxQueueEvents: 0. SQLite and in-memory retention behavior is unchanged.

Broker failover timing

PostgreSQL uses the database clock for processing leases, broker heartbeats, and stale-session takeover. leaseDurationMs defaults to 30,000 ms and drives the background coordination cadence:

Operation	Formula	Default
Broker heartbeat	`max(1000, floor(leaseDurationMs / 3))`	10 s
Duplicate broker-ID stale takeover	`max(leaseDurationMs, 3 × heartbeat interval)`	30 s
Expired processing-lease scan	`min(15000, max(500, floor(leaseDurationMs / 2)))`	15 s

A worker’s lockDuration still determines when its processing generation expires. A surviving broker normally recovers that generation on the next recovery scan, so detection can take the remaining processing lease plus up to one scan interval, before database and scheduler latency. A replacement process reusing the same brokerId must wait for the stale-takeover window; a different unique broker ID can start immediately. Lowering the coordination duration also increases heartbeat and recovery traffic, so tune it from measured database latency and pause behavior rather than desired failover time alone. The scan is capped at 15 s so that a long leaseDurationMs never delays the recovery of a shorter lock TTL or job timeout.

Long values are honoured as configured. Every interval above, pollIntervalMs (durable-event and cron polling), and long-poll waits keep their full length even beyond about 24.8 days, the longest delay a single JavaScript timer accepts. Retries after a transient failure (post-commit maintenance, projection and queue refreshes) follow pollIntervalMs but wait at most 1 s. Both leaseDurationMs and pollIntervalMs accept up to 9007199254740991 ms (Number.MAX_SAFE_INTEGER). statementTimeoutMs, lockTimeoutMs, and idleTransactionTimeoutMs accept 1 to 2147483647 ms, PostgreSQL’s own upper limit for those settings: 0, a negative or sub-millisecond value is rejected, because a 1 ms statement_timeout would fail nearly every statement. A value out of range stops startup with an error that names the setting, whether it comes from the environment, the config file or programmatic configuration. Lease and poll values below their runtime minimum are still raised to that minimum.

PostgreSQL 18 performance settings

bunqueue does not override server-wide PostgreSQL settings. PostgreSQL 18 uses the worker asynchronous I/O method by default, but this queue workload is normally dominated by cached B-tree lookups, WAL, and row coordination rather than cold sequential reads. Measure the actual server before increasing io_workers or io_max_concurrency; more workers can add overhead when the working set is already cached.

For a dedicated database, size shared_buffers from the available memory and working set (PostgreSQL documents roughly 25% of system memory as a starting point, not a universal target). wal_compression=lz4 can reduce full-page-image WAL at a CPU cost. Keep fsync=on, synchronous_commit=on, and full_page_writes=on when acknowledged queue writes must survive a database or host crash. Disabling those durability controls is not a bunqueue performance mode.

Profile with pg_stat_statements, track_io_timing, pg_stat_io, and pg_stat_wal. In the native 20,000-job/16-consumer engineering profile, the final indexed claim path produced zero temp files and read almost entirely from shared buffers. Its final rates were 11,749 admission, 9,782 processing, and 5,338 complete lifecycle jobs/s with two managers and 16 claim loops. A 21-instance PostgreSQL 18.6 matrix found only a 6.3% median uplift from a 512 MiB shared buffer allocation and no decisive AIO/JIT/WAL compression winner. Treat those figures as local diagnostics and benchmark the production storage, payload size, broker count, and journal retention window.

After the commit-ordered journal was finalized, a fresh 10,000-job profile with the same two-manager/16-loop shape measured 11,318 admission, 10,064 processing, and 5,327 lifecycle jobs/s. Moving the commit token to a compact envelope removed the second event-row rewrite and its roughly 49 MiB of profiled WAL per 10,000-job run. These remain local engineering diagnostics, not production sizing claims.

Connection security and deadlines

Use a PostgreSQL URL with the SSL mode required by your provider, for example ?sslmode=verify-full; install the provider CA in the host trust store and do not downgrade certificate verification in production. Bun passes PostgreSQL runtime parameters when each pooled connection opens: bunqueue defaults to a 30-second statement timeout, 5-second lock timeout, and 30-second idle transaction timeout. Set stricter values only after measuring the longest valid queue maintenance operation. Schema migration uses the same lock deadline, so a busy rollout fails safely instead of waiting indefinitely and can be retried.

The Bun SQL pool also uses a 10-second connection timeout, closes connections after 30 seconds idle, and rotates connections after a maximum lifetime of 3,600 seconds. These lifecycle values are fixed runtime safeguards rather than public configuration fields. Include the connection timeout in failover expectations and ensure the provider, proxy, and DNS behavior can reconnect within the application’s retry policy.

The default pool is four connections per broker. Budget total database connections as brokers × poolSize, plus administration, monitoring, and failover headroom. bunqueue also admits 16 active and 128 queued PostgreSQL operations per broker by default. Once that bounded queue is full, commands fail fast and callers may retry with jitter instead of consuming unbounded process memory during an outage.

Schema upgrades, mixed versions, and rollback

Schema initialization is automatic and protected by a PostgreSQL advisory lock, but mixed bunqueue binary versions are not a supported steady state. The safe upgrade procedure is:

Verify backup/PITR and test the target bunqueue version against a restored clone.
Drain or stop all old brokers before the first new binary initializes the database.
Start one new broker and wait for /ready; verify schema health and authoritative counts.
Start the remaining brokers at the same bunqueue version, then update clients.

The initializer refuses a database whose recorded schema version is newer than the binary supports. Consequently, after a migration, restarting an old binary may fail and an application-only downgrade is not a rollback plan. Roll forward, or restore the pre-upgrade database/PITR point together with the old binaries. PostgreSQL schema v21 adds exact event-retention state, transaction-private deltas, and guarded statement-level insert/delete triggers; it therefore requires the same all-brokers-together upgrade procedure. If those derived retention tables drift, initialization locks event writes, repairs their exact primary keys, rebuilds counts from the journal, and restores the triggers in one transaction. The current migrations are additive, but that does not by itself certify every pair of releases for zero-downtime mixed-version operation; validate the exact source/target pair on a clone if continuous availability is mandatory.

Vacuum, failover, and recovery drills

Queue churn creates dead tuples in jobs, events, commit envelopes, completions, metrics, and logs. Keep autovacuum enabled, monitor n_dead_tup, vacuum lag, table/index growth, transaction age, WAL generation, and replica replay lag, and tune per-table autovacuum thresholds from observed churn. Adaptive journal GC does not replace vacuum.

Use PostgreSQL physical or managed-service backups with PITR. Test restores and primary promotion regularly: restore to a fresh cluster, start one broker, verify schema/health and authoritative counts, then add the remaining brokers. DNS or proxy failover must preserve the same database and namespace. A database restored to an earlier point can legitimately replay jobs whose later ACK was not part of that recovery point, so processors must remain idempotent and use application-level deduplication for external effects.

SQLite retention and schema upgrades

maxCompletedJobs is a hot-cache/recovery bound, not a database-retention policy. queue.clean(..., 'completed') is SQLite-authoritative and deletes the oldest eligible retained rows even after they leave that cache. For automatic retention, set storage.completedRetentionMs, BUNQUEUE_COMPLETED_RETENTION_MS, or --completed-retention-ms; it is disabled by default and removes at most 1,000 rows per cleanup tick. Live dependency consumers protect the completed rows and results they still require. Completed-only queues remain visible in queue listings while any durable row exists, including after restart; the cleanup tick unregisters the name after the last row is committed away. Invalid direct retention values cannot make that deletion immediate: negative, non-finite, and unsafe values disable the policy, while finite non-negative fractions are floored to milliseconds. In the server’s own configuration (config file, environment or CLI flag) such a value is rejected instead, and the server does not start.

Deleting retained rows makes their SQLite pages reusable, which bounds future growth for a steady workload, but it does not automatically shrink a database file that is already large. To return that existing free space to the filesystem, stop bunqueue and run SQLite VACUUM in a maintenance window. The operation rewrites the database, so provision enough temporary free disk space and take a backup first.

SQLite schema upgrades run before TCP and HTTP listeners bind. Startup logs the source/target schema versions, database size, each migration step, periodic row/byte progress for legacy payload rewrites, completion duration, and a structured failure if a step cannot finish. The legacy name backfills commit at most 500 rows or 8 MiB of source payload per transaction (one larger row is processed alone) and checkpoint the cursor in the same commit. After interruption, restart with the same or a newer bunqueue version to continue from that checkpoint.

Do not downgrade an in-place, partially or fully migrated database: once a payload-rewrite batch commits, an older binary may no longer understand those rows. Roll forward, or restore the pre-upgrade backup together with the old binary. A database whose recorded schema is newer than the running binary is rejected before startup configuration can mutate it. Because listeners bind only after migration and recovery, this process does not serve HTTP 503 during the upgrade; use process logs and your supervisor’s readiness state until the server starts listening.

The rest of this page covers hosts where the disk does not survive restarts.

Deploying without a durable local disk

Option 1: Mount a persistent volume (simplest)

Most container platforms can attach a durable disk. Point the data path at it and SQLite behaves normally across restarts.

Platform	Durable storage
Fly.io	Fly Volumes
Railway	Volumes
Render	Persistent Disks
Docker / Compose	A named volume mounted at the data directory
Kubernetes	A PersistentVolumeClaim

Option 2: Store-and-forward with a persistent local spool

When the uplink can fail but the instance has a persistent local volume, run bunqueue embedded and forward jobs to one central, durable bunqueue server:

const local = new Queue('ingest', {
  embedded: true,
  dataPath: '/var/lib/bunqueue/spool.db',
  defaultJobOptions: { durable: true }, // close SQLite's 10ms hard-crash window
});

const forwarder = local.forward({
  to: { host: 'central.internal', port: 6789, tls: true },
  queue: 'ingest', // optional remote queue name
});

If the central server is unreachable, jobs stay local and retry; permanent failures land in the local DLQ (dead letter queue, the holding area for jobs that exhausted their retries). This protects network outages while the local process and volume survive. A /tmp spool on a scale-to-zero instance is not durable: if no volume can be attached, write directly to the central server instead. Full walkthrough: IoT & Edge.

Option 3: One central server, stateless workers

Run a single bunqueue server on a host with a durable disk. Producers and workers connect over TCP and hold no state themselves:

const queue = new Queue('jobs', { connection: { host: 'queue.internal', port: 6789 } });
const worker = new Worker('jobs', processor, {
  connection: { host: 'queue.internal', port: 6789 },
});

import { Queue, Worker } from 'bunqueue-client';

const connection = { host: 'queue.internal', port: 6789 };
const queue = new Queue('jobs', { connection });
const worker = new Worker('jobs', processor, { connection });

from bunqueue import Queue, Worker

queue = Queue("jobs", host="queue.internal", port=6789)
worker = Worker("jobs", process, host="queue.internal", port=6789)
worker.run()

use Bunqueue\Queue;
use Bunqueue\Worker;

$queue = new Queue('jobs', ['host' => 'queue.internal', 'port' => 6789]);
$worker = new Worker('jobs', $processor, ['host' => 'queue.internal', 'port' => 6789]);
$worker->run();

queue := bunqueue.NewQueue("jobs", bunqueue.Options{Host: "queue.internal", Port: 6789})

worker := bunqueue.NewWorker("jobs", processor, bunqueue.WorkerOptions{
    Host: "queue.internal",
    Port: 6789,
})
worker.Run()

use bunqueue_client::{ConnectionOptions, Queue, Worker, WorkerOptions};

let connection = ConnectionOptions {
    host: "queue.internal".into(),
    port: 6789,
    ..Default::default()
};
let queue = Queue::new("jobs", connection.clone());
let worker = Worker::new("jobs", processor, WorkerOptions { connection, ..Default::default() });
worker.run()?;

queue = Bunqueue.queue("jobs", host: "queue.internal", port: 6789)

worker = Bunqueue.Worker.new("jobs", processor, host: "queue.internal", port: 6789)
Bunqueue.Worker.run(worker)

Clients exist for Node.js, Deno, Python, PHP, Go, Rust, Elixir, and Cloudflare Workers, see SDKs.

Choosing

Your situation	Use
Container with an attachable disk	Persistent volume (Option 1)
Intermittent uplink plus a persistent local volume	Store-and-forward (Option 2)
Scale-to-zero instance with no durable local disk	Direct central server (Option 3)
Many stateless workers, one durable host	Central TCP server (Option 3)
Multiple active bunqueue brokers sharing state	PostgreSQL 15–18 backend

Guarantees and boundaries

Capability	Memory / SQLite	PostgreSQL 15–18
Embedded mode	Yes	No
Standalone server	Yes	Yes
Active brokers sharing one queue	One	Multiple
Authority	Process memory + optional SQLite durability	PostgreSQL transactions
Claim coordination	In-process locks	Row locks + `SKIP LOCKED`
Lease clock	Broker process	PostgreSQL
Job-group ordering and capacity	In-process; SQLite persists config/order	Transactional across brokers
Group pause/manual deadline	Pause persists; live deadline is in memory	Both persist in PostgreSQL
Built-in S3 snapshots	SQLite only	No; use database backups
MySQL compatibility	No	No

The existing SQLite performance figures apply only to SQLite. PostgreSQL functional validation is not a benchmark, and its throughput depends on network, database sizing, connection pool, retention, and transaction latency.