Skip to content
bunqueue bunqueue
Get started
Get started
Memory/SQLite Domain Layer: Sharding, Queues & States
View Markdown
architecture · domain layer

The domain layer, no I/O.

The memory/SQLite domain layer contains pure queue logic: no I/O, just core algorithms and data structures. PostgreSQL shares the public job model but owns ordering and coordination in database transactions.

This page describes the base memory/SQLite engine. PostgreSQL servers use the same public states and payload types, but authoritative queues, limits, leases, and claims live in PostgreSQL rather than these in-memory shards. See Storage backends and the application layer.

Module Structure

Section titled “Module Structure”
src/domain/
├── types/           # Type definitions
└── queue/           # Core queue logic
    ├── shard.ts             # Shard container
    ├── priorityQueue.ts     # 4-ary indexed heap
    ├── dlqShard.ts          # Dead letter queue
    ├── uniqueKeyManager.ts  # Deduplication
    ├── limiterManager.ts    # Rate/concurrency
    ├── groupLimiterManager.ts # Per-group rate/concurrency
    ├── groupScheduler.ts    # Secondary priority/FIFO lanes + rotation
    ├── dependencyTracker.ts # Job dependencies
    ├── temporalManager.ts   # Temporal index + delayed jobs
    ├── waiterManager.ts     # Long-poll waiters
    └── shardCounters.ts     # Running shard totals

Sharding Architecture

Section titled “Sharding Architecture”

In memory/SQLite mode, jobs are distributed across N shards (auto-detected from CPU cores) for parallelism:

QueueManagerN independent shards, auto-detected
queueName
→
fnv1a()
→
& SHARD_MASK
→
idx
↓
Shard 0 queues, unique, dlq, limits
Shard 1 queues, unique, dlq, limits
Shard 2 queues, unique, dlq, limits
Shard N queues, unique, dlq, limits

Shard count is a power of 2, based on CPU cores, max 64.

Shard Composition

Section titled “Shard Composition”

Each shard is a composition of managers:

Shardcomposition of managers
queues Map<string, PriorityQueue>
UniqueKeyManager deduplication with TTL
DlqShard failed job storage
LimiterManager rate and concurrency control
DependencyTracker waitingDeps + dependencyIndex
TemporalManager delayed jobs, MinHeap
stats running shard totals: queued, delayed, dlq
group ownership active set + authoritative counts
waiters queue-local cursor deques, long-poll support

The shard counters make aggregate queued, delayed, and DLQ totals constant-time. Splitting ready jobs into waiting versus prioritized still examines current queue entries. Multi-queue summary calls batch that work and traverse global processing/completed/dependency collections once, instead of once per queue.

Priority Queue Flow

Section titled “Priority Queue Flow”

4-ary indexed heap with lazy deletion:

PriorityQueue4-ary indexed heap with lazy deletion
PUSH
1. Generate generation number, 2. add to index Map<jobId, {job, generation}>, 3. push to heap {jobId, priority, runAt, generation}, 4. bubbleUp O(log₄ n)
POP
Loop: 1. peek heap top, 2. check index for matching generation, 3. if generation mismatch, stale entry: removeTop, continue, 4. if match: removeTop, delete from index, return job O(log₄ n) amortized
REMOVE, by jobId
1. Delete from index O(1), 2. heap entry becomes stale skipped on pop, 3. compact heap when stale ratio > 20%

Long-Poll Waiters

Section titled “Long-Poll Waiters”

Waiters are isolated by queue. Each queue keeps an append-only entry array with a head cursor, an active count, and one coalesced pending-notification bit. Notification clears a waiter’s timer immediately and advances the cursor; it does not repeatedly filter or splice the full array. Consumed prefixes are compacted once the head reaches 1,024 entries and at least half the array is stale. Surplus batch notifications collapse into one retry hint rather than accumulating credits that would cause repeated empty pulls.

Job State Machine

Section titled “Job State Machine”
Job state machine
WAITING re-entered when a retryable fail triggers retry
↓
DELAYED delay > 0, becomes ready when runAt is reached
ready delay = 0
↓
ACTIVE on retryable fail, back to WAITING
↓
COMPLETED success
DLQ fail at max retries, or timeout

Dependency Resolution Flow

Section titled “Dependency Resolution Flow”
Dependency resolutionJob B, dependsOn: [A]
push B, job with dependencies
1. Push B, check: is A completed?
↓
NO add B to waitingDeps, register B in dependencyIndex[A]
YES push B to active queue
when A completes
1. Add A.id to pendingDepChecks
↓
2. Event-driven flush scheduled on the next microtask, coalescing completions from the same tick; a 30s interval acts as safety fallback only
↓
3. For each completedId, get dependencyIndex[completedId] Set<jobIds>
↓
4. For each waiting job, check all deps in completedJobs, if YES move from waitingDeps to queue

Reverse Index:

Reverse indexdependencyIndex: Map<JobId, Set<JobId>>
A
→
{B, C} B and C wait for A
D
→
{E} E waits for D

DLQ (Dead Letter Queue) Flow

Section titled “DLQ (Dead Letter Queue) Flow”
Move to DLQjob fails with attempts >= maxAttempts
DlqEntry
job original job
reason explicit_fail, max_attempts_exceeded, timeout, stalled, ttl_expired, worker_lost, unknown
error error message
attempts full history: attempt, error, duration
enteredAt timestamp
nextRetryAt if autoRetry enabled
expiresAt 7 days default
↓
DLQ maintenance, every 60s
1. Auto-retry eligible entries nextRetryAt <= now && retryCount < maxAutoRetries
2. Purge expired entries expiresAt <= now
3. Enforce maxEntries per queue 10k default, FIFO eviction when full

Rate & Concurrency Limiting

Section titled “Rate & Concurrency Limiting”
Pull requestrate and concurrency limiting
1. check rate limit, token bucket
Tokens available consume 1, proceed
No tokens return null
↓
2. check concurrency limit
active < limit increment, proceed
At limit return null
↓
3. Pop from priority queue
token bucket
capacity N tokens
refillRate N tokens/sec
tryAcquire() 1. refill based on elapsed time, 2. if tokens >= 1 consume and return true, 3. else return false

FIFO Groups

Section titled “FIFO Groups”

Groups preserve claim order within each group without making group execution serial by default. Active ownership is counted: activeGroupCounts is the authoritative per-group count, while activeGroups is its set-shaped view for telemetry and membership. With no group concurrency option, the limit is unbounded. A Worker can supply a default per-group concurrency cap, and an explicit server-side override can replace it for one group. Per-group fixed window rate limits are checked by the same eligibility path.

Priority/FIFO groupssecondary lanes over the authoritative queue
PULL
1. Promote due secondary entries, 2. serve ready ungrouped work first, 3. otherwise inspect the next group in circular rotation
↓
Ineligible group keep its priority/FIFO head in place and rotate to another group
Eligible group claim its priority/FIFO head, increment ownership, advance the round-robin cursor
ACK / FAIL
1. Decrement the authoritative ownership count, 2. remove set membership only when the count reaches zero, 3. notify waiting pulls

The primary priority queue remains authoritative. GroupScheduler is a lazy secondary view built only when grouped work appears: one heap for ready ungrouped jobs, one delayed/TTL heap, and one immutable priority/FIFO lane per group. Lower BullMQ Pro group priorities run first; equal-priority entries keep their durable admission order. These indexes let a rate- or concurrency-blocked group remain parked while other groups continue round-robin, avoiding queue-head blocking and temporary pop/reinsert cycles. Primary and secondary membership change together under the same synchronous shard lock.