guide · stall-detection

Stuck jobs come back.

If a worker crashes or hangs mid-job, the job is not lost. bunqueue notices the silence, retries the job, and parks repeat offenders in the dead letter queue.

30s+ of heartbeat silence marks a job stalled (confirmed over two 5s sweeps, so ~35–40s in practice)3 stalls before a job moves to the DLQ5s grace period after a job starts

While a worker processes a job it sends periodic heartbeats, small “I’m still alive” signals. If heartbeats stop (crashed process, hung code, dead network), the job is stalled: bunqueue re-queues it for another worker, and after too many stalls moves it to the dead letter queue (DLQ), the holding area for jobs that keep failing.

Stall detection is on by default with sensible defaults. A job may run for hours without stalling as long as automatic heartbeats continue. Tune these thresholds when one uninterrupted work segment can block heartbeats for more than 30 seconds, or when you need a different recovery budget. Detection is poll-driven and two-phase: a job must exceed stallInterval in two consecutive 5-second sweeps before it is marked stalled, so the earliest detection is roughly stallInterval plus one to two sweeps (~35–40s at the defaults).

Configuration

Bun

import { Queue } from 'bunqueue/client';

const queue = new Queue('my-queue', { embedded: true });

queue.setStallConfig({
  enabled: true,         // on by default
  stallInterval: 30000,  // stalled after 30s without a heartbeat
  maxStalls: 3,          // move to DLQ after 3 stalls
  gracePeriod: 5000,     // no stall checks in the first 5s of a job
});

Node.js / Deno

import { Queue } from 'bunqueue-client';

const queue = new Queue('my-queue', { embedded: false });

await queue.setStallConfigAsync({
  enabled: true,         // on by default
  stallInterval: 30000,  // stalled after 30s without a heartbeat
  maxStalls: 3,          // move to DLQ after 3 stalls
  gracePeriod: 5000,     // no stall checks in the first 5s of a job
});

Python

from bunqueue import Queue

queue = Queue("my-queue")

queue.set_stall_config({
    "enabled": True,         # on by default
    "stallInterval": 30000,  # stalled after 30s without a heartbeat
    "maxStalls": 3,          # move to DLQ after 3 stalls
    "gracePeriod": 5000,     # no stall checks in the first 5s of a job
})

PHP

The PHP SDK does not expose this broker command yet. Set the same queue policy through the HTTP API:

curl -X PUT http://localhost:6790/queues/my-queue/stall-config \
  -H 'content-type: application/json' \
  -d '{"config":{"enabled":true,"stallInterval":30000,"maxStalls":3,"gracePeriod":5000}}'

Go

The Go SDK does not expose this broker command yet. Set the same queue policy through the HTTP API:

curl -X PUT http://localhost:6790/queues/my-queue/stall-config \
  -H 'content-type: application/json' \
  -d '{"config":{"enabled":true,"stallInterval":30000,"maxStalls":3,"gracePeriod":5000}}'

Rust

The Rust SDK does not expose this broker command yet. Set the same queue policy through the HTTP API:

curl -X PUT http://localhost:6790/queues/my-queue/stall-config \
  -H 'content-type: application/json' \
  -d '{"config":{"enabled":true,"stallInterval":30000,"maxStalls":3,"gracePeriod":5000}}'

Elixir

The Elixir SDK does not expose this broker command yet. Set the same queue policy through the HTTP API:

curl -X PUT http://localhost:6790/queues/my-queue/stall-config \
  -H 'content-type: application/json' \
  -d '{"config":{"enabled":true,"stallInterval":30000,"maxStalls":3,"gracePeriod":5000}}'

The stall policy is server-side state per queue: a policy set from any client (or the HTTP API) governs jobs processed by workers in every language. The stall-config helper ships in the Bun, TypeScript, and Python clients; the PHP, Go, Rust, and Elixir SDKs do not expose one yet.

Option	Default	Description
enabled	true	Enable/disable stall detection
stallInterval	30000	Time (ms) without a heartbeat before a job is stalled
maxStalls	3	Max stalls before moving to DLQ
gracePeriod	5000	Initial grace period (ms) after a job starts

On the worker side, heartbeats are automatic:

Bun

const worker = new Worker('queue', processor, {
  embedded: true,
  heartbeatInterval: 10000, // heartbeat every 10 seconds (default)
});

Node.js / Deno

const worker = new Worker('queue', processor, {
  embedded: false,
  heartbeatInterval: 10000, // heartbeat every 10 seconds (default)
});

Python

worker = Worker("queue", process, heartbeat_interval_s=10.0)  # default; 0 disables

PHP

$worker = new Worker('queue', $processor, [
    'heartbeatIntervalS' => 10.0, // Fires between jobs (sequential worker)
]);

Go

worker := bunqueue.NewWorker("queue", processor, bunqueue.WorkerOptions{
    HeartbeatIntervalS: 10, // Heartbeats are disabled by default in Go
})

Rust

let worker = Worker::new("queue", processor, WorkerOptions {
    heartbeat_interval: Some(Duration::from_secs(10)), // None disables
    ..Default::default()
});

Elixir

worker = Bunqueue.Worker.new("queue", handler, heartbeat_interval: 10_000)  # ms

Keep heartbeatInterval well below stallInterval, otherwise healthy jobs get flagged as stalled.

With SQLite or PostgreSQL persistence enabled, a custom stall policy and every job’s cumulative stall count survive process and broker restarts. A crash consumes one attempts slot and one stallCount slot; reaching either maxAttempts or maxStalls is terminal and moves the job to the DLQ. Repeated crashes therefore cannot reset either retry budget.

Long-running jobs

A long total runtime is safe with heartbeats. Use a wider stall window when a single processing segment can block the runtime or network long enough to miss the default 30-second window:

Bun

// Video processing may take hours
const videoQueue = new Queue('video-processing', { embedded: true });

videoQueue.setStallConfig({
  stallInterval: 300000,  // 5 minutes
  maxStalls: 2,
  gracePeriod: 60000,
});

const worker = new Worker('video-processing', async (job) => {
  for (const chunk of video.chunks) {
    await processChunk(chunk);
    await job.updateProgress(chunk.progress); // also counts as a heartbeat
  }
}, { embedded: true, heartbeatInterval: 30000 });

Node.js / Deno

// Video processing may take hours
const videoQueue = new Queue('video-processing', { embedded: false });

await videoQueue.setStallConfigAsync({
  stallInterval: 300000,  // 5 minutes
  maxStalls: 2,
  gracePeriod: 60000,
});

const worker = new Worker('video-processing', async (job) => {
  for (const chunk of video.chunks) {
    await processChunk(chunk);
    await job.updateProgress(chunk.progress); // also counts as a heartbeat
  }
}, { embedded: false, heartbeatInterval: 30000 });

Python

# Video processing may take hours
video_queue = Queue("video-processing")

video_queue.set_stall_config({
    "stallInterval": 300000,  # 5 minutes
    "maxStalls": 2,
    "gracePeriod": 60000,
})

def process(job):
    for chunk in video.chunks:
        process_chunk(chunk)
        job.update_progress(chunk.progress)  # also counts as a heartbeat

worker = Worker("video-processing", process, heartbeat_interval_s=30.0)

PHP

Configure the five-minute broker policy through the HTTP API shown above, then report progress from the sequential handler so it also refreshes the stall timer:

$worker = new Worker('video-processing', function (Bunqueue\Job $job) use ($video) {
    foreach ($video->chunks as $chunk) {
        processChunk($chunk);
        $job->updateProgress($chunk->progress);
    }
}, ['lockTtlMs' => 300000]);

Go

Configure the five-minute broker policy through the HTTP API shown above. The worker heartbeat and progress updates both refresh liveness:

worker := bunqueue.NewWorker("video-processing", func(job *bunqueue.Job) (any, error) {
    for _, chunk := range video.Chunks {
        processChunk(chunk)
        if err := job.UpdateProgress(chunk.Progress, ""); err != nil { return nil, err }
    }
    return nil, nil
}, bunqueue.WorkerOptions{HeartbeatIntervalS: 30, LockTtlMs: 300_000})

Rust

Configure the five-minute broker policy through the HTTP API shown above. The worker heartbeat and progress updates both refresh liveness:

let worker = Worker::new("video-processing", move |job| {
    for chunk in &video.chunks {
        process_chunk(chunk);
        job.update_progress(chunk.progress, None)
            .map_err(|error| ProcessError::retryable(error.to_string()))?;
    }
    Ok(Value::Nil)
}, WorkerOptions {
    heartbeat_interval: Some(Duration::from_secs(30)),
    lock_ttl_ms: 300_000,
    ..Default::default()
});

Elixir

Configure the five-minute broker policy through the HTTP API shown above. The worker heartbeat and progress updates both refresh liveness:

handler = fn job ->
  Enum.each(video.chunks, fn chunk ->
    process_chunk(chunk)
    {:ok, _} = Bunqueue.Job.update_progress(job, chunk.progress)
  end)
  {:ok, nil}
end

worker =
  Bunqueue.Worker.new("video-processing", handler,
    heartbeat_interval: 30_000, lock_ttl: 300_000
  )

Two things reset the stall timer: the worker’s automatic heartbeat (every heartbeatInterval ms) and any job.updateProgress() call. For long jobs without natural progress points, the automatic heartbeat is enough.

What happens when a job stalls

1. Retry: the path depends on how the stall was detected. Heartbeat-stall recovery re-queues the job with its stall count incremented and runAt pushed out by the job’s exponential backoff, without waking blocked pullers, so pickup waits for the backoff plus the next poll. Lock-expiry recovery re-queues without backoff and notifies waiting workers immediately.
2. DLQ: the job becomes terminal when either its cumulative stall count reaches maxStalls or the interrupted delivery consumes its final normal attempts slot. The attempts check wins ties: since each stall also consumes an attempt, with the defaults (maxStalls: 3, attempts: 3) a repeatedly stalling job lands in the DLQ as max_attempts_exceeded. The stalled classification appears only when maxStalls is lower than the job’s remaining attempts budget.

Listening for stalls

The Bun package can listen through embedded or TCP QueueEvents:

Bun

import { QueueEvents } from 'bunqueue/client';

const events = new QueueEvents('my-queue', {
  embedded: false,
  connection: { host: '127.0.0.1', port: 6789 },
});
await events.waitUntilReady();
events.on('stalled', ({ jobId }) => {
  console.log(`Job ${jobId} stalled`);
});

Node.js / Deno

import { QueueEvents } from 'bunqueue-client';

const events = new QueueEvents('my-queue', {
  embedded: false,
  connection: { host: '127.0.0.1', port: 6789 },
});
await events.waitUntilReady();
events.on('stalled', ({ jobId }) => {
  console.log(`Job ${jobId} stalled`);
});

Python

The network SDK does not receive broker-side stall events. Subscribe to the queue’s SSE stream and handle frames whose SSE event name is job:stalled (the JSON data carries queue, jobId, timestamp):

curl -N http://localhost:6790/events/queues/my-queue

PHP

The network SDK does not receive broker-side stall events. Subscribe to the queue’s SSE stream and handle frames whose SSE event name is job:stalled (the JSON data carries queue, jobId, timestamp):

curl -N http://localhost:6790/events/queues/my-queue

Go

The network SDK does not receive broker-side stall events. Subscribe to the queue’s SSE stream and handle frames whose SSE event name is job:stalled (the JSON data carries queue, jobId, timestamp):

curl -N http://localhost:6790/events/queues/my-queue

Rust

The network SDK does not receive broker-side stall events. Subscribe to the queue’s SSE stream and handle frames whose SSE event name is job:stalled (the JSON data carries queue, jobId, timestamp):

curl -N http://localhost:6790/events/queues/my-queue

Elixir

The network SDK does not receive broker-side stall events. Subscribe to the queue’s SSE stream and handle frames whose SSE event name is job:stalled (the JSON data carries queue, jobId, timestamp):

curl -N http://localhost:6790/events/queues/my-queue

The shared TypeScript Worker also emits stalled in embedded and TCP modes. Its TCP path uses the same dedicated authenticated broker subscription as QueueEvents and re-subscribes after reconnect. The other language SDKs should use SSE or WebSocket for this broker-side event. Stall webhooks are not emitted, so do not register job.stalled as a webhook event. This notification is preserved when an expired lease consumes the final maxStalls or maxAttempts slot: the broker publishes stalled before the terminal failed queue event and moves the job to the DLQ.

Monitoring

Bun

const stats = queue.getDlqStats();
console.log('Stalled jobs in DLQ:', stats.byReason.stalled);

const stalledJobs = queue.getDlq({ reason: 'stalled' });

Node.js / Deno

const stats = await queue.getDlqStatsAsync();
console.log('Stalled jobs in DLQ:', stats.byReason.stalled);

const stalledJobs = await queue.getDlqAsync({ reason: 'stalled' });

Python

# get_dlq() returns raw jobs without a `reason` field.
# Use the Bun client to filter by reason.
jobs = queue.get_dlq()

PHP

// getDlq() returns raw jobs without a `reason` field.
// Use the Bun client to filter by reason.
$jobs = $queue->getDlq();

Go

// GetDlq returns raw jobs without a `reason` field.
// Use the Bun client to filter by reason.
jobs, err := queue.GetDlq(0) // 0 means no explicit count bound

Rust

// get_dlq returns raw jobs without a `reason` field.
// Use the Bun client to filter by reason.
let jobs = queue.get_dlq(None)?;

Elixir

# dlq/1 returns raw jobs without a `reason` field.
# Use the Bun client to filter by reason.
{:ok, jobs} = Bunqueue.Queue.dlq(queue)

Both TypeScript packages expose authoritative getDlqStatsAsync() and getDlqAsync({ reason }) over TCP. The other SDKs list raw DLQ jobs with getDlq(count?) / get_dlq() / GetDlq(count) and do not expose the entry’s failure-reason metadata through those helpers.

SandboxedWorker

Experimental

SandboxedWorker depends on experimental Bun Workers. For production, use the standard Worker. See Worker vs SandboxedWorker.

SandboxedWorker also sends heartbeats automatically in both modes; in embedded mode heartbeatInterval defaults to 5000 ms. If its jobs run longer than stallInterval, either raise stallInterval, call progress() periodically, or disable stall detection with queue.setStallConfig({ enabled: false }).

Related Guides

• Dead Letter Queue - Where stalled jobs end up after max stalls
• Worker API - Configure heartbeat intervals
• CPU-Intensive Workers - Prevent stalls in CPU-heavy workloads