Architecture: SQLite and PostgreSQL Queue Engines for Bun

architecture · overview

Inside the bunqueue architecture.

bunqueue has two execution topologies behind one server protocol: a synchronous sharded memory/SQLite engine, and a database-authoritative PostgreSQL 15–18 engine for multiple brokers. This section maps both and identifies which diagrams belong to which path.

System Overview

System overviewclient, server, persistence

client layer

Queue.add() TcpPool

→

Worker.process() TcpPool

↓ msgpack over TCP :6789

server layer

QueueManager · memory / SQLite

N shards auto-detected: Shard 0, Shard 1, ... Shard N

jobIndex

completedJobs

customIdMap

jobResults

↓

local persistence path

WriteBuffer

→

SQLite WAL mode

PostgreSQL multi-broker path

PostgresQueueManager

→

Bun.SQL pool + transactions

→

PostgreSQL authoritative state

background tasks

Scheduler

Stall detection

DLQ maintenance

Cleanup

Layered Architecture

Layer	Purpose	Key Components
Client	SDK for applications	Queue, Worker, FlowProducer, TcpPool
Server	Request handling	TcpServer, HttpServer, Handlers
Application	Orchestration	QueueManager, Operations, Managers
Domain	Business logic	Shard, PriorityQueue, DLQ
Infrastructure	Storage and external systems	SQLite, PostgreSQL, S3 Backup, Scheduler
Shared	Utilities	Hash, Lock, LRU, MinHeap

Architecture Sections

Section	Description
Client SDK	TCP connection, job submission, worker processing
Domain Layer	Sharding, priority queues, DLQ logic
Application Layer	Operations flow, background tasks
Persistence	SQLite configuration, write buffering, and recovery
Storage Backends	PostgreSQL transactions, multi-broker fencing, and topology
Data Structures	Core algorithms and complexities
TCP Protocol	Wire format and commands
Cron Scheduler	Event-driven scheduling, timezone support, persistence

Key Design Decisions

The shard, heap, lock, write-buffer, and complexity sections below describe the memory/SQLite QueueManager. PostgreSQL servers select PostgresQueueManager instead: PostgreSQL owns claim ordering, leases, shared policy, dependencies, events, cron, and terminal state. The TCP/HTTP client contract remains common.

Dynamic Shard Architecture

In memory/SQLite mode, jobs are distributed across N independent shards (auto-detected from CPU cores) using FNV-1a hash:

SHARD_COUNT = calculateShardCount()  // Power of 2, based on CPU cores, max 64
SHARD_MASK = SHARD_COUNT - 1
shardIndex = fnv1a(queueName) & SHARD_MASK  // src/shared/hash.ts

// Examples: 4 cores → 4 shards, 10 cores → 16 shards, 64+ cores → 64 shards

Benefits:

Auto-scales with hardware (power of 2, max 64)
Parallel operations on different queues
Reduced lock contention
Bitwise AND faster than modulo

4-ary Priority Queue

Each shard contains a 4-ary heap instead of binary:

Better cache locality (children fit in cache line)
Fewer tree levels (8 vs 16 for 65k items)
O(log₄ n) operations

SQLite Write Buffer

Jobs batch before SQLite write:

Buffer 100 jobs

→

Multi-row INSERT 186,384 jobs/s median in the published public on-disk addBulk workload

Flushes after 10ms or when 100 jobs are buffered, whichever comes first.

Buffered: up to 10 ms loss risk; 186,384 jobs/s median in the published public on-disk Embedded addBulk workload
Durable: immediate persistence; 60,835 ops/s median for published sequential Embedded adds

Lazy Deletion

Heap entries use generation tracking:

Remove: Delete from index (O(1)), mark heap entry stale
Pop: Skip entries where generation != current
Compact: Rebuild when >20% stale

Lock Hierarchy

Acquire in order to prevent deadlocks:

1. jobIndex (read-only)
2. completedJobs (check before lock)
3. shardLocks[N]
4. processingLocks[N]

Memory Bounds

Collection	Limit	Eviction
completedJobs	50,000	FIFO batch
jobResults	10,000	LRU
jobLogs	10,000	LRU
customIdMap	50,000	LRU
DLQ per queue	10,000	FIFO

Memory/SQLite Performance Summary

Operation	Complexity
PUSH	O(log₄ n)
PULL	O(log₄ n)
ACK	O(1)
ACK batch	O(shards)
Job lookup	O(1)
Stats	O(1)