Skip to content
Engineering

Background Jobs and Queues in Node.js: BullMQ, pg-boss and Trigger.dev

AIDToolStack EditorialNovember 7, 20269 min read
Node.jsBackground JobsBullMQRedisPostgres
Share:Twitter

The signup handler creates the user, sends a welcome email, resizes the avatar and pushes the contact to the CRM. The CRM API is slow today, so signups take eight seconds, and when the deploy restarts the process halfway through, some users never get their email and nobody knows which ones.

That's the case for a job queue. The request does the one thing that has to happen before responding (write the user), records the rest as jobs, and returns. Separate worker processes pick the jobs up, retry the ones that fail, and leave a record of the ones that never succeed. This post covers how that works in Node, the three options I'd look at in November 2026, and the settings that bite in production.

What a queue gives you that setTimeout doesn't

Firing off a promise and not awaiting it ("fire and forget") looks like a background job and isn't one. If the process exits, the work is gone. A real queue gives you:

  • Durability. The job is stored (in Redis or Postgres) before the request returns.
  • Retries with backoff. Transient failures fix themselves; permanent ones end up somewhere visible.
  • Concurrency control. Ten workers, five jobs each, no matter how many requests arrive.
  • Scheduling. Run later, or on a cron pattern.
  • A place to look. Counts of waiting, active, failed and completed jobs.
Job lifecycle: a job scheduler or an API call adds a job, which is waiting until a worker takes it and makes it active; if the handler returns it is completed, if it throws it becomes delayed for a backoff of 2^(attempt - 1) times the delay and then waits again; after the last attempt or an UnrecoverableError it is failed. With Redis, maxmemory-policy must be noeviction
Job lifecycle: a job scheduler or an API call adds a job, which is waiting until a worker takes it and makes it active; if the handler returns it is completed, if it throws it becomes delayed for a backoff of 2^(attempt - 1) times the delay and then waits again; after the last attempt or an UnrecoverableError it is failed. With Redis, maxmemory-policy must be noeviction

The state names in that diagram are BullMQ's, but every queue has the same shape.

Cron or a queue?

Both, usually. Cron answers when; a queue answers how to do it reliably. A crontab line that runs a 40-minute script has no retries, no concurrency limit and no record of partial failure. The better pattern is a scheduler that enqueues a job, and a worker that runs it with the same retry and monitoring as everything else. BullMQ calls these Job Schedulers, pg-boss has cron scheduling built in, and Trigger.dev has scheduled tasks defined with schedules.task(). If you keep plain cron for anything, the cron best practices post covers overlap, time zones and silent failures, and our cron parser explains five-field expressions and lists their next run times.

The options in November 2026

BullMQpg-bossTrigger.dev
Latest (npm, 7 Nov 2026)6.7.212.38.1@trigger.dev/sdk 4.8.3
StorageRedis (default) or PostgresPostgres 13+Managed service, or self-hosted
Runs your codeYour worker processesYour worker processesTrigger.dev's workers (or your self-hosted ones)
Good fitHigh throughput, Redis already in the stackOne database, jobs created in the same transaction as your dataLong-running tasks and AI agents without running workers yourself

There are others (Graphile Worker on Postgres, the managed queues from each cloud), but these three cover most Node teams.

BullMQ

BullMQ started out as a Redis-backed queue, and version 6 changed its shape. The changelog entry for 6.0.0 (30 July 2026) introduces "Redis and PostgreSQL backends", makes ioredis "an optional peer dependency" rather than a direct one, and removes the legacy repeat option in favour of Job Schedulers. If you're upgrading from v5, install ioredis explicitly and read the migration guide before touching repeatable jobs.

Here's a worker with retries, a custom job ID and a non-retryable error. I ran this against Redis 8.10 in Docker with BullMQ 6.7.2 and ioredis 6.0.0:

import { Queue, Worker, UnrecoverableError } from 'bullmq';
import IORedis from 'ioredis';

const connection = new IORedis({ port: 16479, maxRetriesPerRequest: null });
const queue = new Queue('emails', { connection });

const t0 = Date.now();
const worker = new Worker('emails', async (job) => {
  console.log(`attempt ${job.attemptsMade + 1} for ${job.id} at +${Date.now() - t0}ms`);
  if (job.data.bad) throw new UnrecoverableError('bad address');
  if (job.attemptsMade < 2) throw new Error('SMTP timeout');
  return 'sent';
}, { connection, concurrency: 5 });

worker.on('completed', (job, rv) => console.log(`completed ${job.id}: ${rv}`));
worker.on('failed', (job, err) => console.log(`failed ${job.id} (attempt ${job.attemptsMade}): ${err.message}`));

const opts = { attempts: 4, backoff: { type: 'exponential', delay: 1000 } };
const a = await queue.add('welcome', { userId: 42 }, { ...opts, jobId: 'welcome-42' });
const b = await queue.add('welcome', { userId: 42 }, { ...opts, jobId: 'welcome-42' });
console.log('same job returned twice:', a.id === b.id);
await queue.add('welcome', { userId: 7, bad: true }, { ...opts, jobId: 'welcome-7' });
try { await queue.add('x', {}, { jobId: '123' }); } catch (e) { console.log('numeric id:', e.message); }

The output (trimmed of a count I printed alongside):

attempt 1 for welcome-42 at +8ms
same job returned twice: true
failed welcome-42 (attempt 1): SMTP timeout
numeric id: Custom Id cannot be integers
attempt 1 for welcome-7 at +13ms
failed welcome-7 (attempt 1): bad address
attempt 2 for welcome-42 at +1015ms
failed welcome-42 (attempt 2): SMTP timeout
attempt 3 for welcome-42 at +3090ms
completed welcome-42: sent

Three things in there are worth knowing. The retries landed at about +1 s and +3 s, matching the documented backoff formula of 2 ^ (attempts - 1) * delay (1,000 ms, then 2,000 ms). The same docs offer a jitter option between 0 and 1, which you want when many jobs fail at once. Adding a job whose ID already exists did nothing: per the job ID docs, it "will just be ignored and not added to the queue at all", which makes a business key like welcome-42 a cheap dedupe. Those docs also say custom IDs can't be all digits or contain :, and that an ID becomes reusable once the old job is removed. And UnrecoverableError failed the job on the first attempt instead of burning retries on an address that will never work.

Schedules use the same queue. Calling upsertJobScheduler twice with the same ID left one scheduler in my test, with the next run computed in the given time zone:

await queue.upsertJobScheduler('nightly-report',
  { pattern: '15 3 * * *', tz: 'Europe/London' },
  { name: 'build-report', data: {} });
// getJobSchedulers() -> 1 scheduler, next: 2026-11-08T03:15:00.000Z

BullMQ in production

The going to production guide is short and every point matters:

  • maxmemory-policy noeviction. BullMQ "cannot work properly if Redis evicts keys arbitrarily". If you share a Redis that's configured as an LRU cache, jobs can vanish. BullMQ does warn you; with the policy set to allkeys-lru, adding a job printed IMPORTANT! Eviction policy is allkeys-lru. It should be "noeviction". Give the queue its own Redis instance or database server rather than sharing a cache. The Redis vs Valkey vs Memcached post covers the server choice; the docs say the Redis advice also applies to compatible servers such as Valkey and Dragonfly.
  • maxRetriesPerRequest: null on ioredis connections used by Workers, as in the quick-start.
  • Close workers on shutdown. Call await worker.close() on SIGTERM and SIGINT "to minimize the risk of stalled jobs", and add handlers for uncaughtException and unhandledRejection.
  • Persistence. The guide recommends AOF, noting that fsync once per second "is enough for most applications".

When you're debugging a live queue, the official Redis MCP server lets an AI assistant inspect keys and list lengths; connect it as a read-only ACL user, as its README advises.

Staying on Postgres: pg-boss or BullMQ's new backend

If Postgres is your only datastore, adding Redis just for jobs is a real cost. Postgres has the primitive queues need. The SELECT docs describe SKIP LOCKED as unsuitable for general use but say it "can be used to avoid lock contention with multiple consumers accessing a queue-like table".

pg-boss is built on exactly that. Its README lists Node 22.12 or newer (or Bun) and PostgreSQL 13 or newer, cron and RRULE scheduling, dead letter queues, exponential backoff, LISTEN/NOTIFY for low-latency delivery, and the feature I'd pick it for: you can "create jobs in an existing db transaction", with adapters for Drizzle, Knex, Kysely and Prisma. Creating the job in the same transaction as the row it's about means you never have a user without their welcome email job, or a job for a user whose insert rolled back. The README's basic usage:

const { PgBoss } = require('pg-boss');
const boss = new PgBoss('postgres://user:pass@host/database');
boss.on('error', console.error);
await boss.start();
await boss.createQueue('readme-queue');
const id = await boss.send('readme-queue', { arg1: 'read me' });
await boss.work('readme-queue', async ([job]) => { /* handle job */ });

I didn't run pg-boss for this post (no Postgres on hand); that snippet is from the README.

BullMQ's PostgreSQL backend is the other route: same Queue and Worker API, pass createPostgresBackend, install pg. It needs PostgreSQL 13 or newer (14+ recommended), uses LISTEN/NOTIFY, doesn't run migrations automatically (you call runMigrations as a deploy step) and doesn't support schema downgrades. The docs are candid that Redis "remains the default and the most battle-tested option" and estimate roughly 1.5 to 2 times fewer jobs per second on Postgres, from indicative laptop numbers. It's young; I'd use it for moderate volumes where one fewer service matters more than throughput.

Trigger.dev

Trigger.dev takes the workers off your hands. You write tasks as functions, deploy them, and trigger them from your app. Per its docs, v4 is current and the v2 and v3 docs "describe superseded versions". From the tasks overview, retries are configured per task, and tasks "can execute indefinitely" by default unless you set maxDuration:

import { task } from "@trigger.dev/sdk";

export const taskWithRetries = task({
  id: "task-with-retries",
  retry: { maxAttempts: 10, factor: 1.8, minTimeoutInMs: 500, maxTimeoutInMs: 30_000, randomize: false },
  run: async (payload: { userId: string }) => { /* ... */ },
});

Its idempotency keys dedupe triggers, are stored for 30 days by default, and can be scoped to a run, an attempt or globally. The platform core is Apache-2.0 and self-hostable. The catch is the obvious one: your job code runs on someone else's infrastructure unless you self-host. There's also a Trigger.dev MCP server built into its CLI; run it with --dev-only or --readonly unless you want an agent able to deploy. I didn't run the Trigger.dev example, which needs an account.

Assume every job can run twice

A worker can finish the work and crash before it marks the job complete. Then the job runs again. Design for it:

  • Make handlers idempotent. Check before acting ("is this email already sent for this user?"), or use upserts.
  • Pass idempotency keys downstream. If the job charges a card through Stripe, derive the key from the job's business ID, not a fresh UUID per attempt. The Stripe integration patterns post shows this with plan-change:${id}.
  • Use deterministic job IDs when the same event might enqueue twice.
  • Classify errors. Retry timeouts and 5xx responses; fail fast on validation errors and 4xx.

CPU-heavy work is a different problem

Queues move work out of the request. They don't make CPU-bound JavaScript stop blocking the event loop. The Node docs are clear about where threads help: workers "are useful for performing CPU-intensive JavaScript operations. They do not help much with I/O-intensive work." Image resizing or PDF rendering in a job worker still blocks that worker's other jobs; run those jobs with lower concurrency or a dedicated queue, or hand the heavy step to a worker thread or native library.

Before you ship a queue

  • Jobs are enqueued after (or inside) the transaction that creates their data, never before.
  • Every job has attempts, exponential backoff with jitter, and a terminal failed state someone looks at.
  • Permanent failures throw a non-retryable error.
  • Handlers are idempotent; external calls carry idempotency keys.
  • Redis-backed queues run on noeviction with persistence, on an instance that isn't your cache.
  • Workers close cleanly on SIGTERM, and deploys give them time to.
  • Scheduled work goes through the queue, not straight from crontab.
  • Someone gets alerted when the failed count grows, and structured logs carry the job ID.
AE

Written by AIDToolStack Editorial

Researched from official documentation, specifications and vendor pages, listed under Sources below. Last checked November 2026.

Sources

  1. github.com/taskforcesh/bullmq/blob/master/docs/gitbook/changelog.md
  2. github.com/taskforcesh/bullmq/releases
  3. docs.bullmq.io/quick-start
  4. docs.bullmq.io/guide/going-to-production
  5. docs.bullmq.io/guide/retrying-failing-jobs
  6. docs.bullmq.io/guide/jobs/job-ids
  7. docs.bullmq.io/guide/job-schedulers
  8. docs.bullmq.io/guide/postgresql
  9. www.npmjs.com/package/bullmq
  10. github.com/timgit/pg-boss
  11. www.npmjs.com/package/pg-boss
  12. www.postgresql.org/docs/current/sql-select.html
  13. trigger.dev/docs/introduction
  14. trigger.dev/docs/tasks/overview
  15. trigger.dev/docs/tasks/scheduled
  16. trigger.dev/docs/idempotency
  17. www.npmjs.com/package/@trigger.dev/sdk
  18. nodejs.org/api/worker_threads.html

Related Posts

Explore on AIDToolStack

Discover tools and servers featured in our directory.

MCP Servers

Trigger.dev Jobs MCP ServerManage Trigger.dev background jobs from your AI coding assistant. Monitor job runs, trigger tasks manually, inspect execution logs, manage schedules, and debug failed runs. The server provides operational visibility into your background job infrastructure — see which tasks are running, queued, or failed, and take action. Your AI can help diagnose job failures from execution logs, retry failed tasks, and optimize concurrency configurations for better throughput.Tailscale Network MCP ServerManage your Tailscale mesh VPN network from your AI coding assistant. View connected devices, manage ACLs, configure exit nodes, check device status, and manage DNS settings. The server provides visibility into your Tailscale network — see which machines are online, troubleshoot connectivity issues, manage network access policies, and configure routes. Useful for developers managing distributed infrastructure where Tailscale provides secure networking between services.Portainer Docker MCP ServerManage Docker environments through Portainer's API from your AI coding assistant. Control containers, images, networks, and volumes across multiple Docker hosts and Kubernetes clusters. The server provides tools for container lifecycle management, image pulling and building, stack deployment, and resource monitoring. Portainer's multi-environment management makes this server particularly valuable for teams managing containers across development, staging, and production environments.

Reviews

Sign in to leave a review.

No reviews yet. Be the first!

Comments

Sign in to leave a comment.

No comments yet. Be the first!

Sponsor Our Newsletter

Reach thousands of developers who are actively evaluating AI tools, MCP servers, and dev infrastructure. Our weekly newsletter goes to engaged technical decision-makers.

Growing subscriber baseHigh open rates

All sponsored content is clearly labeled per our editorial policy.