# Stall Detection: Auto-Recover Unresponsive Jobs bunqueue stall detection auto-recovers stuck jobs. It is on by default; tune heartbeat intervals, max stall thresholds, and grace periods for long jobs. Canonical: https://bunqueue.dev/guide/stall-detection/ --- import { Tabs, TabItem } from '@astrojs/starlight/components';
guide · stall-detection

Stuck jobs come back.

If a worker crashes or hangs mid-job, the job is not lost. bunqueue notices the silence, retries the job, and parks repeat offenders in the dead letter queue.

30s+ of heartbeat silence marks a job stalled (confirmed over two 5s sweeps, so ~35–40s in practice) 3 stalls before a job moves to the DLQ 5s grace period after a job starts
While a worker processes a job it sends periodic **heartbeats**, small "I'm still alive" signals. If heartbeats stop (crashed process, hung code, dead network), the job is **stalled**: bunqueue re-queues it for another worker, and after too many stalls moves it to the [dead letter queue](/guide/dlq/) (DLQ), the holding area for jobs that keep failing. **Stall detection is on by default with sensible defaults.** A job may run for hours without stalling as long as automatic heartbeats continue. Tune these thresholds when one uninterrupted work segment can block heartbeats for more than 30 seconds, or when you need a different recovery budget. Detection is poll-driven and two-phase: a job must exceed `stallInterval` in two consecutive 5-second sweeps before it is marked stalled, so the earliest detection is roughly `stallInterval` plus one to two sweeps (~35–40s at the defaults). ## Configuration ```typescript import { Queue } from 'bunqueue/client'; const queue = new Queue('my-queue', { embedded: true }); queue.setStallConfig({ enabled: true, // on by default stallInterval: 30000, // stalled after 30s without a heartbeat maxStalls: 3, // move to DLQ after 3 stalls gracePeriod: 5000, // no stall checks in the first 5s of a job }); ``` ```typescript import { Queue } from 'bunqueue-client'; const queue = new Queue('my-queue', { embedded: false }); await queue.setStallConfigAsync({ enabled: true, // on by default stallInterval: 30000, // stalled after 30s without a heartbeat maxStalls: 3, // move to DLQ after 3 stalls gracePeriod: 5000, // no stall checks in the first 5s of a job }); ``` ```python from bunqueue import Queue queue = Queue("my-queue") queue.set_stall_config({ "enabled": True, # on by default "stallInterval": 30000, # stalled after 30s without a heartbeat "maxStalls": 3, # move to DLQ after 3 stalls "gracePeriod": 5000, # no stall checks in the first 5s of a job }) ``` The PHP SDK does not expose this broker command yet. Set the same queue policy through the HTTP API: ```bash curl -X PUT http://localhost:6790/queues/my-queue/stall-config \ -H 'content-type: application/json' \ -d '{"config":{"enabled":true,"stallInterval":30000,"maxStalls":3,"gracePeriod":5000}}' ``` The Go SDK does not expose this broker command yet. Set the same queue policy through the HTTP API: ```bash curl -X PUT http://localhost:6790/queues/my-queue/stall-config \ -H 'content-type: application/json' \ -d '{"config":{"enabled":true,"stallInterval":30000,"maxStalls":3,"gracePeriod":5000}}' ``` The Rust SDK does not expose this broker command yet. Set the same queue policy through the HTTP API: ```bash curl -X PUT http://localhost:6790/queues/my-queue/stall-config \ -H 'content-type: application/json' \ -d '{"config":{"enabled":true,"stallInterval":30000,"maxStalls":3,"gracePeriod":5000}}' ``` The Elixir SDK does not expose this broker command yet. Set the same queue policy through the HTTP API: ```bash curl -X PUT http://localhost:6790/queues/my-queue/stall-config \ -H 'content-type: application/json' \ -d '{"config":{"enabled":true,"stallInterval":30000,"maxStalls":3,"gracePeriod":5000}}' ``` *The stall policy is server-side state per queue: a policy set from any client (or the HTTP API) governs jobs processed by workers in every language. The stall-config helper ships in the Bun, TypeScript, and Python clients; the PHP, Go, Rust, and Elixir SDKs do not expose one yet.* | Option | Default | Description | |--------|---------|-------------| | `enabled` | `true` | Enable/disable stall detection | | `stallInterval` | `30000` | Time (ms) without a heartbeat before a job is stalled | | `maxStalls` | `3` | Max stalls before moving to DLQ | | `gracePeriod` | `5000` | Initial grace period (ms) after a job starts | On the worker side, heartbeats are automatic: ```typescript const worker = new Worker('queue', processor, { embedded: true, heartbeatInterval: 10000, // heartbeat every 10 seconds (default) }); ``` ```typescript const worker = new Worker('queue', processor, { embedded: false, heartbeatInterval: 10000, // heartbeat every 10 seconds (default) }); ``` ```python worker = Worker("queue", process, heartbeat_interval_s=10.0) # default; 0 disables ``` ```php $worker = new Worker('queue', $processor, [ 'heartbeatIntervalS' => 10.0, // Fires between jobs (sequential worker) ]); ``` ```go worker := bunqueue.NewWorker("queue", processor, bunqueue.WorkerOptions{ HeartbeatIntervalS: 10, // Heartbeats are disabled by default in Go }) ``` ```rust let worker = Worker::new("queue", processor, WorkerOptions { heartbeat_interval: Some(Duration::from_secs(10)), // None disables ..Default::default() }); ``` ```elixir worker = Bunqueue.Worker.new("queue", handler, heartbeat_interval: 10_000) # ms ``` Keep `heartbeatInterval` well below `stallInterval`, otherwise healthy jobs get flagged as stalled. With SQLite or PostgreSQL persistence enabled, a custom stall policy and every job's cumulative stall count survive process and broker restarts. A crash consumes one `attempts` slot and one `stallCount` slot; reaching either `maxAttempts` or `maxStalls` is terminal and moves the job to the DLQ. Repeated crashes therefore cannot reset either retry budget. ## Long-running jobs A long total runtime is safe with heartbeats. Use a wider stall window when a single processing segment can block the runtime or network long enough to miss the default 30-second window: ```typescript // Video processing may take hours const videoQueue = new Queue('video-processing', { embedded: true }); videoQueue.setStallConfig({ stallInterval: 300000, // 5 minutes maxStalls: 2, gracePeriod: 60000, }); const worker = new Worker('video-processing', async (job) => { for (const chunk of video.chunks) { await processChunk(chunk); await job.updateProgress(chunk.progress); // also counts as a heartbeat } }, { embedded: true, heartbeatInterval: 30000 }); ``` ```typescript // Video processing may take hours const videoQueue = new Queue('video-processing', { embedded: false }); await videoQueue.setStallConfigAsync({ stallInterval: 300000, // 5 minutes maxStalls: 2, gracePeriod: 60000, }); const worker = new Worker('video-processing', async (job) => { for (const chunk of video.chunks) { await processChunk(chunk); await job.updateProgress(chunk.progress); // also counts as a heartbeat } }, { embedded: false, heartbeatInterval: 30000 }); ``` ```python # Video processing may take hours video_queue = Queue("video-processing") video_queue.set_stall_config({ "stallInterval": 300000, # 5 minutes "maxStalls": 2, "gracePeriod": 60000, }) def process(job): for chunk in video.chunks: process_chunk(chunk) job.update_progress(chunk.progress) # also counts as a heartbeat worker = Worker("video-processing", process, heartbeat_interval_s=30.0) ``` Configure the five-minute broker policy through the HTTP API shown above, then report progress from the sequential handler so it also refreshes the stall timer: ```php $worker = new Worker('video-processing', function (Bunqueue\Job $job) use ($video) { foreach ($video->chunks as $chunk) { processChunk($chunk); $job->updateProgress($chunk->progress); } }, ['lockTtlMs' => 300000]); ``` Configure the five-minute broker policy through the HTTP API shown above. The worker heartbeat and progress updates both refresh liveness: ```go worker := bunqueue.NewWorker("video-processing", func(job *bunqueue.Job) (any, error) { for _, chunk := range video.Chunks { processChunk(chunk) if err := job.UpdateProgress(chunk.Progress, ""); err != nil { return nil, err } } return nil, nil }, bunqueue.WorkerOptions{HeartbeatIntervalS: 30, LockTtlMs: 300_000}) ``` Configure the five-minute broker policy through the HTTP API shown above. The worker heartbeat and progress updates both refresh liveness: ```rust let worker = Worker::new("video-processing", move |job| { for chunk in &video.chunks { process_chunk(chunk); job.update_progress(chunk.progress, None) .map_err(|error| ProcessError::retryable(error.to_string()))?; } Ok(Value::Nil) }, WorkerOptions { heartbeat_interval: Some(Duration::from_secs(30)), lock_ttl_ms: 300_000, ..Default::default() }); ``` Configure the five-minute broker policy through the HTTP API shown above. The worker heartbeat and progress updates both refresh liveness: ```elixir handler = fn job -> Enum.each(video.chunks, fn chunk -> process_chunk(chunk) {:ok, _} = Bunqueue.Job.update_progress(job, chunk.progress) end) {:ok, nil} end worker = Bunqueue.Worker.new("video-processing", handler, heartbeat_interval: 30_000, lock_ttl: 300_000 ) ``` Two things reset the stall timer: the worker's automatic heartbeat (every `heartbeatInterval` ms) and any `job.updateProgress()` call. For long jobs without natural progress points, the automatic heartbeat is enough. ## What happens when a job stalls 1. **Retry**: the path depends on how the stall was detected. Heartbeat-stall recovery re-queues the job with its stall count incremented and `runAt` pushed out by the job's exponential backoff, without waking blocked pullers, so pickup waits for the backoff plus the next poll. Lock-expiry recovery re-queues without backoff and notifies waiting workers immediately. 2. **DLQ**: the job becomes terminal when either its cumulative stall count reaches `maxStalls` or the interrupted delivery consumes its final normal `attempts` slot. The attempts check wins ties: since each stall also consumes an attempt, with the defaults (`maxStalls: 3`, `attempts: 3`) a repeatedly stalling job lands in the DLQ as `max_attempts_exceeded`. The `stalled` classification appears only when `maxStalls` is lower than the job's remaining `attempts` budget. ## Listening for stalls The Bun package can listen through embedded or TCP `QueueEvents`: ```typescript import { QueueEvents } from 'bunqueue/client'; const events = new QueueEvents('my-queue', { embedded: false, connection: { host: '127.0.0.1', port: 6789 }, }); await events.waitUntilReady(); events.on('stalled', ({ jobId }) => { console.log(`Job ${jobId} stalled`); }); ``` ```typescript import { QueueEvents } from 'bunqueue-client'; const events = new QueueEvents('my-queue', { embedded: false, connection: { host: '127.0.0.1', port: 6789 }, }); await events.waitUntilReady(); events.on('stalled', ({ jobId }) => { console.log(`Job ${jobId} stalled`); }); ``` The network SDK does not receive broker-side stall events. Subscribe to the queue's SSE stream and handle frames whose SSE `event` name is `job:stalled` (the JSON `data` carries `queue`, `jobId`, `timestamp`): ```bash curl -N http://localhost:6790/events/queues/my-queue ``` The network SDK does not receive broker-side stall events. Subscribe to the queue's SSE stream and handle frames whose SSE `event` name is `job:stalled` (the JSON `data` carries `queue`, `jobId`, `timestamp`): ```bash curl -N http://localhost:6790/events/queues/my-queue ``` The network SDK does not receive broker-side stall events. Subscribe to the queue's SSE stream and handle frames whose SSE `event` name is `job:stalled` (the JSON `data` carries `queue`, `jobId`, `timestamp`): ```bash curl -N http://localhost:6790/events/queues/my-queue ``` The network SDK does not receive broker-side stall events. Subscribe to the queue's SSE stream and handle frames whose SSE `event` name is `job:stalled` (the JSON `data` carries `queue`, `jobId`, `timestamp`): ```bash curl -N http://localhost:6790/events/queues/my-queue ``` The network SDK does not receive broker-side stall events. Subscribe to the queue's SSE stream and handle frames whose SSE `event` name is `job:stalled` (the JSON `data` carries `queue`, `jobId`, `timestamp`): ```bash curl -N http://localhost:6790/events/queues/my-queue ``` The shared TypeScript `Worker` also emits `stalled` in embedded and TCP modes. Its TCP path uses the same dedicated authenticated broker subscription as QueueEvents and re-subscribes after reconnect. The other language SDKs should use SSE or WebSocket for this broker-side event. Stall webhooks are not emitted, so do not register `job.stalled` as a webhook event. This notification is preserved when an expired lease consumes the final `maxStalls` or `maxAttempts` slot: the broker publishes `stalled` before the terminal `failed` queue event and moves the job to the DLQ. ## Monitoring ```typescript const stats = queue.getDlqStats(); console.log('Stalled jobs in DLQ:', stats.byReason.stalled); const stalledJobs = queue.getDlq({ reason: 'stalled' }); ``` ```typescript const stats = await queue.getDlqStatsAsync(); console.log('Stalled jobs in DLQ:', stats.byReason.stalled); const stalledJobs = await queue.getDlqAsync({ reason: 'stalled' }); ``` ```python # get_dlq() returns raw jobs without a `reason` field. # Use the Bun client to filter by reason. jobs = queue.get_dlq() ``` ```php // getDlq() returns raw jobs without a `reason` field. // Use the Bun client to filter by reason. $jobs = $queue->getDlq(); ``` ```go // GetDlq returns raw jobs without a `reason` field. // Use the Bun client to filter by reason. jobs, err := queue.GetDlq(0) // 0 means no explicit count bound ``` ```rust // get_dlq returns raw jobs without a `reason` field. // Use the Bun client to filter by reason. let jobs = queue.get_dlq(None)?; ``` ```elixir # dlq/1 returns raw jobs without a `reason` field. # Use the Bun client to filter by reason. {:ok, jobs} = Bunqueue.Queue.dlq(queue) ``` *Both TypeScript packages expose authoritative `getDlqStatsAsync()` and `getDlqAsync({ reason })` over TCP. The other SDKs list raw DLQ jobs with `getDlq(count?)` / `get_dlq()` / `GetDlq(count)` and do not expose the entry's failure-reason metadata through those helpers.* ## SandboxedWorker :::caution[Experimental] `SandboxedWorker` depends on experimental Bun Workers. For production, use the standard `Worker`. See [Worker vs SandboxedWorker](/guide/worker/sandboxed/#worker-vs-sandboxedworker). ::: `SandboxedWorker` also sends heartbeats automatically in both modes; in embedded mode `heartbeatInterval` defaults to `5000` ms. If its jobs run longer than `stallInterval`, either raise `stallInterval`, call `progress()` periodically, or disable stall detection with `queue.setStallConfig({ enabled: false })`. :::tip[Related Guides] - [Dead Letter Queue](/guide/dlq/) - Where stalled jobs end up after max stalls - [Worker API](/guide/worker/) - Configure heartbeat intervals - [CPU-Intensive Workers](/guide/cpu-intensive-workers/) - Prevent stalls in CPU-heavy workloads :::