guide · queue

The same job, only once.

Retried HTTP calls, at-least-once webhooks and impatient users all produce duplicate adds. A custom id or a dedup key makes the second one a no-op instead of a second charge.

Deduplication

With jobId (idempotent adds)

Give a job a custom jobId and adding it twice does nothing: while a generation with that ID is live (waiting, delayed, prioritized, waiting-children, or active), the existing job is returned instead of creating a duplicate. Custom IDs are broker-wide because they are also the global persisted job primary key, so this remains idempotent even if the second add targets another queue. Once the prior generation is terminal, the ID may be reused; bunqueue retires its completed/DLQ state before admitting exactly one fresh generation. This makes add() safe to call repeatedly in embedded and TCP modes.

Bun

const job1 = await queue.add('process', { orderId: 123 }, { jobId: 'order-123' });
const job2 = await queue.add('process', { orderId: 123 }, { jobId: 'order-123' });

console.log(job1.id === job2.id); // true, same job returned

Node.js / Deno

const job1 = await queue.add('process', { orderId: 123 }, { jobId: 'order-123' });
const job2 = await queue.add('process', { orderId: 123 }, { jobId: 'order-123' });

console.log(job1.id === job2.id); // true, same job returned

Python

job1 = queue.add("process", {"order_id": 123}, job_id="order-123")
job2 = queue.add("process", {"order_id": 123}, job_id="order-123")

print(job1.id == job2.id)  # True, same job returned

PHP

$job1 = $queue->add('process', ['orderId' => 123], ['jobId' => 'order-123']);
$job2 = $queue->add('process', ['orderId' => 123], ['jobId' => 'order-123']);

var_dump($job1->id() === $job2->id()); // true, same job returned

Go

job1, _ := queue.Add("process", map[string]any{"orderId": 123}, bunqueue.JobOptions{"jobId": "order-123"})
job2, _ := queue.Add("process", map[string]any{"orderId": 123}, bunqueue.JobOptions{"jobId": "order-123"})

fmt.Println(job1.ID() == job2.ID()) // true, same job returned

Rust

let opts = || JobOptions { job_id: Some("order-123".into()), ..Default::default() };
let job1 = queue.add("process", data.clone(), opts())?;
let job2 = queue.add("process", data, opts())?;

assert_eq!(job1.id(), job2.id()); // same job returned

Elixir

{:ok, job1} = Bunqueue.Queue.add(queue, "process", %{order_id: 123}, jobId: "order-123")
{:ok, job2} = Bunqueue.Queue.add(queue, "process", %{order_id: 123}, jobId: "order-123")

job1.id == job2.id  # true, same job returned

Typical uses: webhook retries, double-submits from a UI, restoring jobs on service startup without duplicating them.

Completed ids are reused, not returned

Idempotency collapses onto every unfinished generation, including a job that is currently active. Only after the previous job completes or fails does re-adding the same jobId start a fresh generation; bunqueue evicts the stale terminal record first. So an id such as report-2026-06-17 is safe to reuse: it is idempotent during one run and starts cleanly after that run becomes terminal.

With a TTL window

The deduplication option dedupes within a time window instead of permanently. The id field is required:

Bun

// Same id within 1 hour = no new job. After the TTL, a new job is allowed.
await queue.add('notification', { userId: '123' }, {
  deduplication: { id: 'notify-123', ttl: 3600000 }
});

Node.js / Deno

// Same id within 1 hour = no new job. After the TTL, a new job is allowed.
await queue.add('notification', { userId: '123' }, {
  deduplication: { id: 'notify-123', ttl: 3600000 }
});

Python

# Same id within 1 hour = no new job. After the TTL, a new job is allowed.
queue.add("notification", {"user_id": "123"},
          deduplication={"id": "notify-123", "ttl": 3600000})

PHP

// Same id within 1 hour = no new job. After the TTL, a new job is allowed.
$queue->add('notification', ['userId' => '123'], [
    'deduplication' => ['id' => 'notify-123', 'ttl' => 3600000],
]);

Go

// Same id within 1 hour = no new job. After the TTL, a new job is allowed.
queue.Add("notification", map[string]any{"userId": "123"}, bunqueue.JobOptions{
    "deduplication": map[string]any{"id": "notify-123", "ttl": 3600000},
})

Rust

use bunqueue_client::{Deduplication, JobOptions};

// Same id within 1 hour = no new job. After the TTL, a new job is allowed.
queue.add("notification", data, JobOptions {
    deduplication: Some(Deduplication {
        id: "notify-123".into(),
        ttl: Some(3_600_000),
        ..Default::default()
    }),
    ..Default::default()
})?;

Elixir

# The Elixir SDK currently supports idempotency through jobId.
# TTL deduplication keys are not exposed yet.
{:ok, _job} =
  Bunqueue.Queue.add(queue, "notification", %{user_id: "123"},
    jobId: "notify-123"
  )

Two strategies change what happens when a duplicate arrives:

Bun

// extend: keep the existing job, reset its TTL (debouncing, "keep quiet while active")
await queue.add('sync-task', { action: 'sync' }, {
  deduplication: { id: 'sync-task', ttl: 60000, extend: true }
});

// replace: remove the pending job, insert a new one with the latest data (last write wins)
await queue.add('latest-data', { data: newData }, {
  deduplication: { id: 'data-job', ttl: 300000, replace: true }
});

Node.js / Deno

// extend: keep the existing job, reset its TTL (debouncing, "keep quiet while active")
await queue.add('sync-task', { action: 'sync' }, {
  deduplication: { id: 'sync-task', ttl: 60000, extend: true }
});

// replace: remove the pending job, insert a new one with the latest data (last write wins)
await queue.add('latest-data', { data: newData }, {
  deduplication: { id: 'data-job', ttl: 300000, replace: true }
});

Python

# extend: keep the existing job, reset its TTL (debouncing)
queue.add("sync-task", {"action": "sync"},
          deduplication={"id": "sync-task", "ttl": 60000, "extend": True})

# replace: remove the pending job, insert a new one with the latest data
queue.add("latest-data", {"data": new_data},
          deduplication={"id": "data-job", "ttl": 300000, "replace": True})

PHP

// extend: keep the existing job, reset its TTL (debouncing)
$queue->add('sync-task', ['action' => 'sync'], [
    'deduplication' => ['id' => 'sync-task', 'ttl' => 60000, 'extend' => true],
]);

// replace: remove the pending job, insert a new one with the latest data
$queue->add('latest-data', ['data' => $newData], [
    'deduplication' => ['id' => 'data-job', 'ttl' => 300000, 'replace' => true],
]);

Go

// extend: keep the existing job, reset its TTL (debouncing)
queue.Add("sync-task", map[string]any{"action": "sync"}, bunqueue.JobOptions{
    "deduplication": map[string]any{"id": "sync-task", "ttl": 60000, "extend": true},
})

// replace: remove the pending job, insert a new one with the latest data
queue.Add("latest-data", map[string]any{"data": newData}, bunqueue.JobOptions{
    "deduplication": map[string]any{"id": "data-job", "ttl": 300000, "replace": true},
})

Rust

// extend: keep the existing job, reset its TTL (debouncing)
queue.add("sync-task", data, JobOptions {
    deduplication: Some(Deduplication {
        id: "sync-task".into(),
        ttl: Some(60_000),
        extend: Some(true),
        ..Default::default()
    }),
    ..Default::default()
})?;

// replace: remove the pending job, insert a new one with the latest data
queue.add("latest-data", new_data, JobOptions {
    deduplication: Some(Deduplication {
        id: "data-job".into(),
        ttl: Some(300_000),
        replace: Some(true),
        ..Default::default()
    }),
    ..Default::default()
})?;

Elixir

# TTL extend/replace strategies are not exposed by the Elixir SDK yet.
# A stable jobId still collapses repeated adds while the job is unfinished.
{:ok, first} = Bunqueue.Queue.add(queue, "sync-task", %{action: "sync"}, jobId: "sync-task")
{:ok, duplicate} = Bunqueue.Queue.add(queue, "sync-task", %{action: "sync"}, jobId: "sync-task")
true = first.id == duplicate.id

Option	Type	Default	Description
id	string	(required)	Unique deduplication key
ttl	number	-	Time in ms before the key expires
extend	boolean	false	Reset TTL on duplicate, keep existing job
replace	boolean	false	Remove pending job, create a new one (new internal id)

Replace details

replace: true never touches a job that is already processing (active). If both extend and replace are set, replace wins.

For durable jobs, replacement is committed as one persistence transition: the superseded pending row is removed before add() resolves and cannot reappear after a broker restart. If several same-key replacements are submitted in one bulk add, only the final generation remains runnable and recoverable.

When the current owner is active, it keeps running under its existing lease. A durable replacement atomically moves persisted key ownership to the queued successor, so an acknowledgement or failure from the older generation cannot release the new generation’s key, including across a restart before that acknowledgement. A non-durable successor retains the normal write-buffer crash window.

Replacement is rejected if another live job depends on the owner’s id. The broker leaves both jobs and their dependency edge unchanged instead of creating a permanently blocked dependent. Change or remove the dependency explicitly before replacing that owner.

addBulk is ordered but is not an all-or-nothing transaction. If a later entry fails, any earlier accepted prefix remains persisted, counted and runnable; entries after the failure are not evaluated. Use the flow API when the complete graph must commit atomically.

Managing keys directly in Bun, or retaining an explicit custom job ID in the network SDKs when later lookup is required:

Bun

const jobId = await queue.getDeduplicationJobId('my-unique-key'); // look up
await queue.removeDeduplicationKey('my-unique-key');              // allow re-adding

const job = await queue.getJob(jobId!);
const removed = await job?.removeDeduplicationKey(); // only if this generation owns it

Node.js / Deno

const job = await queue.add('notification', data, { jobId: 'my-custom-job-id' });
const sameJob = await queue.getJobByCustomId('my-custom-job-id');
console.log(job.id === sameJob?.id); // true

Python

job = queue.add("notification", data, job_id="my-custom-job-id")
same_job = queue.get_job_by_custom_id("my-custom-job-id")
assert same_job is not None and job.id == same_job.id

PHP

// Explicit custom job ids can be looked up; TTL deduplication keys cannot.
$job = $queue->getJobByCustomId('my-custom-job-id');
$jobId = $job?->id();

Go

// Explicit custom job ids can be looked up; TTL deduplication keys cannot.
job, err := queue.GetJobByCustomID("my-custom-job-id")
if err != nil {
    return err
}

Rust

// Explicit custom job ids can be looked up; TTL deduplication keys cannot.
let job = queue.get_job_by_custom_id("my-custom-job-id")?;
let job_id = job.map(|job| job.id());

Elixir

# Explicit custom job ids can be looked up; TTL deduplication keys cannot.
{:ok, job} = Bunqueue.Queue.get_job_by_custom_id(queue, "my-custom-job-id")
job_id = if job, do: job.id, else: nil

The Bun queue lookup/removal methods work in embedded and TCP modes. Job-level removal is generation-safe: a stale job cannot clear a key already transferred to a replacement job. Direct TTL deduplication-key lookup and removal are currently Bun-only. A custom jobId is a separate value: network SDKs can look it up, but that query never resolves deduplication.id.

Where to go next

Queue API	Create a queue in embedded or TCP mode
Adding Jobs	add, addBulk, priorities, delays, durability
Querying Jobs	Fetch jobs, states, counts and results
Queue Control and Maintenance	Pause, drain, obliterate, clean and repair
Progress, Job Logs and Dependencies	Progress, per-job logs and dependencies
Queue Rate Limiting and Global Concurrency	Rate limits and global concurrency caps
Job Schedulers from the Queue	Named repeatable schedules from the queue
DLQ Operations from the Queue Object	Failed-job operations from the Queue object
Workers, Stats and Metrics from the Queue	Registered workers, stats and metrics windows
Namespaces, Auto-Batching and Store-and-Forward	Namespaces, auto-batching, store-and-forward
JobOptions Reference	Every JobOptions field, with defaults