# Memory/SQLite Domain Layer: Sharding, Queues & States bunqueue memory/SQLite domain internals: auto-scaled sharding, 4-ary priority queues, job state machine, DLQ flow, and rate limiting logic. Canonical: https://bunqueue.dev/architecture/domain-layer/ ---
architecture · domain layer

The domain layer, no I/O.

The memory/SQLite domain layer contains pure queue logic: no I/O, just core algorithms and data structures. PostgreSQL shares the public job model but owns ordering and coordination in database transactions.

This page describes the base memory/SQLite engine. PostgreSQL servers use the same public states and payload types, but authoritative queues, limits, leases, and claims live in PostgreSQL rather than these in-memory shards. See [Storage backends](/guide/databases/) and the [application layer](/architecture/application-layer/). ## Module Structure ``` src/domain/ ├── types/ # Type definitions └── queue/ # Core queue logic ├── shard.ts # Shard container ├── priorityQueue.ts # 4-ary indexed heap ├── dlqShard.ts # Dead letter queue ├── uniqueKeyManager.ts # Deduplication ├── limiterManager.ts # Rate/concurrency ├── groupLimiterManager.ts # Per-group rate/concurrency ├── groupScheduler.ts # Secondary priority/FIFO lanes + rotation ├── dependencyTracker.ts # Job dependencies ├── temporalManager.ts # Temporal index + delayed jobs ├── waiterManager.ts # Long-poll waiters └── shardCounters.ts # Running shard totals ``` ## Sharding Architecture In memory/SQLite mode, jobs are distributed across N shards (auto-detected from CPU cores) for parallelism:
QueueManagerN independent shards, auto-detected
queueName
→
fnv1a()
→
& SHARD_MASK
→
idx
↓
Shard 0 queues, unique, dlq, limits
Shard 1 queues, unique, dlq, limits
Shard 2 queues, unique, dlq, limits
Shard N queues, unique, dlq, limits

Shard count is a power of 2, based on CPU cores, max 64.

### Shard Composition Each shard is a composition of managers:
Shardcomposition of managers
queues Map<string, PriorityQueue>
UniqueKeyManager deduplication with TTL
DlqShard failed job storage
LimiterManager rate and concurrency control
DependencyTracker waitingDeps + dependencyIndex
TemporalManager delayed jobs, MinHeap
stats running shard totals: queued, delayed, dlq
group ownership active set + authoritative counts
waiters queue-local cursor deques, long-poll support
The shard counters make aggregate queued, delayed, and DLQ totals constant-time. Splitting ready jobs into `waiting` versus `prioritized` still examines current queue entries. Multi-queue summary calls batch that work and traverse global processing/completed/dependency collections once, instead of once per queue. ## Priority Queue Flow 4-ary indexed heap with lazy deletion:
PriorityQueue4-ary indexed heap with lazy deletion
PUSH
1. Generate generation number, 2. add to index Map<jobId, {job, generation}>, 3. push to heap {jobId, priority, runAt, generation}, 4. bubbleUp O(log₄ n)
POP
Loop: 1. peek heap top, 2. check index for matching generation, 3. if generation mismatch, stale entry: removeTop, continue, 4. if match: removeTop, delete from index, return job O(log₄ n) amortized
REMOVE, by jobId
1. Delete from index O(1), 2. heap entry becomes stale skipped on pop, 3. compact heap when stale ratio > 20%
## Long-Poll Waiters Waiters are isolated by queue. Each queue keeps an append-only entry array with a head cursor, an active count, and one coalesced pending-notification bit. Notification clears a waiter's timer immediately and advances the cursor; it does not repeatedly filter or splice the full array. Consumed prefixes are compacted once the head reaches 1,024 entries and at least half the array is stale. Surplus batch notifications collapse into one retry hint rather than accumulating credits that would cause repeated empty pulls. ## Job State Machine
Job state machine
WAITING re-entered when a retryable fail triggers retry
↓
DELAYED delay > 0, becomes ready when runAt is reached
ready delay = 0
↓
ACTIVE on retryable fail, back to WAITING
↓
COMPLETED success
DLQ fail at max retries, or timeout
## Dependency Resolution Flow
Dependency resolutionJob B, dependsOn: [A]
push B, job with dependencies
1. Push B, check: is A completed?
↓
NO add B to waitingDeps, register B in dependencyIndex[A]
YES push B to active queue
when A completes
1. Add A.id to pendingDepChecks
↓
2. Event-driven flush scheduled on the next microtask, coalescing completions from the same tick; a 30s interval acts as safety fallback only
↓
3. For each completedId, get dependencyIndex[completedId] Set<jobIds>
↓
4. For each waiting job, check all deps in completedJobs, if YES move from waitingDeps to queue
**Reverse Index:**
Reverse indexdependencyIndex: Map<JobId, Set<JobId>>
A
→
{B, C} B and C wait for A
D
→
{E} E waits for D
## DLQ (Dead Letter Queue) Flow
Move to DLQjob fails with attempts >= maxAttempts
DlqEntry
job original job
reason explicit_fail, max_attempts_exceeded, timeout, stalled, ttl_expired, worker_lost, unknown
error error message
attempts full history: attempt, error, duration
enteredAt timestamp
nextRetryAt if autoRetry enabled
expiresAt 7 days default
↓
DLQ maintenance, every 60s
1. Auto-retry eligible entries nextRetryAt <= now && retryCount < maxAutoRetries
2. Purge expired entries expiresAt <= now
3. Enforce maxEntries per queue 10k default, FIFO eviction when full
## Rate & Concurrency Limiting
Pull requestrate and concurrency limiting
1. check rate limit, token bucket
Tokens available consume 1, proceed
No tokens return null
↓
2. check concurrency limit
active < limit increment, proceed
At limit return null
↓
3. Pop from priority queue
token bucket
capacity N tokens
refillRate N tokens/sec
tryAcquire() 1. refill based on elapsed time, 2. if tokens >= 1 consume and return true, 3. else return false
## FIFO Groups Groups preserve claim order within each group without making group execution serial by default. Active ownership is counted: `activeGroupCounts` is the authoritative per-group count, while `activeGroups` is its set-shaped view for telemetry and membership. With no group concurrency option, the limit is unbounded. A Worker can supply a default per-group concurrency cap, and an explicit server-side override can replace it for one group. Per-group fixed window rate limits are checked by the same eligibility path.
Priority/FIFO groupssecondary lanes over the authoritative queue
PULL
1. Promote due secondary entries, 2. serve ready ungrouped work first, 3. otherwise inspect the next group in circular rotation
↓
Ineligible group keep its priority/FIFO head in place and rotate to another group
Eligible group claim its priority/FIFO head, increment ownership, advance the round-robin cursor
ACK / FAIL
1. Decrement the authoritative ownership count, 2. remove set membership only when the count reaches zero, 3. notify waiting pulls
The primary priority queue remains authoritative. `GroupScheduler` is a lazy secondary view built only when grouped work appears: one heap for ready ungrouped jobs, one delayed/TTL heap, and one immutable priority/FIFO lane per group. Lower BullMQ Pro group priorities run first; equal-priority entries keep their durable admission order. These indexes let a rate- or concurrency-blocked group remain parked while other groups continue round-robin, avoiding queue-head blocking and temporary pop/reinsert cycles. Primary and secondary membership change together under the same synchronous shard lock. :::tip[Related] - [Architecture Overview](/architecture/) - Full component map - [Data Structures](/architecture/data-structures/) - The 4-ary MinHeap and skip list behind these queues - [Application Layer](/architecture/application-layer/) - Operations that drive these state transitions :::