Do Not Make the User Wait: A Practical Guide to Queues and Background Jobs, and Why Your Job Will Sometimes Run Twice
The laundry ticket
You drop your clothes at a laundry and the attendant hands you a ticket with a number. You do not wait while the machines run. The shop works in the background, and you come back later or check the ticket.
Now imagine a laundry with no tickets. Every customer stands at the counter until their clothes are done. On a busy Monday, the shop is full of people waiting, and nobody new can even get in.
Many apps are built like the second laundry. A user clicks “Place order,” and the server saves the order, sends an email, generates a PDF, resizes images, calls a partner’s system and updates a report, all before it answers. The user stares at a spinner for something they never needed to wait for.
The big idea is that you should do only what the user must wait for
A simple test for each step in a request: does the user need this result right now to continue?
- Yes: keep it in the request. For example, validating input and saving the order.
- No: put a message on a queue and let a background worker do it. For example, confirmation emails, PDF receipts, image and video processing, webhooks to other systems, search indexing, analytics events, bulk imports and cleanup tasks.
The request then returns quickly with a job ID. An HTTP 202 Accepted response means “received, not finished yet,” and the user can check a status page or get notified later.
A queue does not speed up a single job. What it gives you is survival:
- A shock absorber. A sudden spike of orders piles up in the queue instead of crashing your database.
- Retries. A failed job can be tried again instead of being lost.
- Isolation. A slow partner system delays one background job, not your whole website.
There is a catch that surprises many developers. Most queues promise at least once delivery, not exactly once. Amazon SQS standard queues guarantee that every message is delivered at least once, but in some cases it may be delivered more than once, and the documentation tells you to design your application with idempotent operations. SQS standard queues also provide only best effort ordering. SQS FIFO queues add exactly once processing, with a five minute deduplication interval for repeated sends, at the cost of different throughput characteristics.
So your handler will sometimes run twice. If it sends an email or charges a card, “twice” can hurt.
The reliability toolkit
1. Acknowledge after success, not before. The broker should only delete a job once the worker finishes it. If the worker crashes, the job comes back.
2. Make handlers idempotent. Give every job a unique key. Before doing the work, check whether that key is already done. Better still, make the work and the “done” marker part of the same database transaction where you can.
3. Set the visibility or lock timeout longer than the job takes. With SQS, a message becomes invisible to other consumers for the visibility timeout, 30 seconds by default and up to 12 hours. If your job takes longer, the message reappears and another worker starts a duplicate.
4. Retry with backoff and a limit. Wait longer after each failure, add random jitter, and stop after a fixed number of attempts.
5. Use a dead letter queue. Jobs that keep failing move to a separate queue, so one bad message does not cycle forever or block the others. SQS supports dead letter queues for exactly this, and it is also how you find problem messages. Then monitor and alert on it, because a dead letter queue nobody reads is a trash can.
6. Use separate queues for different kinds of work. A flood of slow report jobs should never delay a password reset email.
7. Set a timeout per job so a stuck job cannot hold a worker forever.
A small pseudo code illustration:
POST /reports
job_id = new_unique_id()
db.save(job_id, status = “queued”)
queue.send({ job_id, user_id })
return 202 Accepted { job_id, status_url = “/jobs/” + job_id }
Get Rufus Kehinde’s stories in your inbox
Join Medium for free to get updates from this writer.
worker:
message = queue.receive()
if db.is_done(message.job_id): # safe to run twice
ack(message); return
do_the_work()
db.mark_done(message.job_id)
ack(message) # acknowledge only after success
The dual write problem, and the outbox
Here is a classic trap. When an order is placed, your code saves it in the database and then publishes an “OrderPlaced” message to the broker. Those are two separate systems. If the app crashes between the two steps, you either lose the message, or, if you publish first, you send a message about an order that never saved.
The standard solution is the transactional outbox:
1. In one database transaction, save the order and also insert a row in an outbox table.
2. A separate relay process reads the outbox and publishes messages to the broker, marking them as sent.
Because the order and the outbox row commit together, either both exist or neither does. It works in any database with transactions, such as PostgreSQL, SQL Server, Oracle and Db2. In MongoDB, use multi-document transactions where available, or change streams.
You can write the relay yourself, or let change data capture do it. Debezium reads the database’s transaction log and provides connectors for PostgreSQL, SQL Server, MongoDB, Db2 and Oracle, among others. It includes an outbox event router for relational databases, and MongoDB has its own separate outbox router, so check the connector’s maturity for your database and version.
Start simple — the database as a queue
For modest volumes, you do not always need a separate broker. A jobs table can work as a queue if workers claim rows safely so two workers never take the same one:
- PostgreSQL and Oracle: SELECT … FOR UPDATE SKIP LOCKED
- SQL Server: UPDLOCK with the READPAST hint
- MongoDB: an atomic findOneAndUpdate that flips a job from “queued” to “processing”
This keeps one system to operate. The cost is more load on your primary database, and you will outgrow it at higher volumes.
Choosing a queue
- Message brokers such as RabbitMQ: messages are delivered to consumers and removed once acknowledged.
- Event logs such as Apache Kafka: messages are kept for a retention period and consumers track their own position, so they can replay history. Ordering is guaranteed within a partition.
- Cloud queues: Amazon SQS and SNS, Azure Service Bus and Google Pub/Sub remove most operations work.
- Redis based job libraries: simple and fast, with a durability tradeoff you must understand.
- A durable workflow engine such as Temporal: for long, multi step processes that must survive failures.
A warning if you use Redis for jobs
BullMQ, a popular Node.js job queue, documents that its Redis must use the noeviction memory policy, because it cannot work properly if Redis evicts keys arbitrarily. If you point a cache with an eviction policy and a job queue at the same Redis, memory pressure can silently delete jobs. Use separate Redis instances for cache and queues. BullMQ also keeps completed and failed jobs forever by default, so set automatic removal, and its job data is stored in clear text, so avoid putting sensitive data in jobs.
In your stack, check each one’s documentation for your version:
- Node.js and NestJS: BullMQ
- Python and Django: Celery, with RabbitMQ or Redis, and RQ
- PHP and Laravel: the built in queues, with Redis, SQS or database drivers, and Horizon for Redis queues
- .NET: Hangfire, MassTransit and Azure Service Bus
- Java and Spring: Spring AMQP, Spring Kafka and Quartz
- Go: Asynq, or Temporal
- Ruby and Rails: Sidekiq
The scale lens
- The backlog arithmetic. If 1 million jobs arrive in a minute and your workers process 500 per second, it takes 2,000 seconds, about 33 minutes, to clear. If jobs arrive faster than workers can process them, the queue grows without limit. Set a target for how long each type of job may wait, and size your workers for it.
- Alert on age, not only depth. The age of the oldest message tells you how long users are waiting. A queue with a modest length but an old oldest message is already hurting people.
- Scale workers on the backlog. Autoscale on queue depth or age, and cap concurrency so you do not overload the database the workers depend on.
- Retry storms. When a downstream system fails, every worker retries at once and makes it worse. Use backoff with jitter and a circuit breaker.
- Fairness. One huge tenant can fill a shared queue and starve everyone else. Use per tenant concurrency limits, priorities or separate queues.
- Ordering and partitions. If order matters, use a key so related messages go to the same partition or message group, and watch for hot partitions where one key gets too much traffic.
- Rare becomes daily. At 10 million jobs per day, a 0.01% duplicate rate is 1,000 duplicate executions every day. (Illustrative numbers.) Idempotency is not optional.
A security note
- Secure the broker. Use authentication, TLS and least privilege per queue, and never expose a broker or a Redis instance to the internet.
- Keep secrets and personal data out of messages. Pass IDs, not whole records, and look up the data when the job runs.
- Validate messages like any other input. Anything that can reach the queue can forge a message. Verify webhook signatures, limit payload sizes, and avoid unsafe deserialization.
- Protect the dead letter queue. It holds the messages that failed, often the most interesting ones, so apply access control and retention limits.
- Limit who can enqueue. An open endpoint that creates jobs can be abused to flood your queue, so rate limit it.
- Audit money related jobs. Keep a record of what ran, when and with which key.
The tradeoffs
- The user sees “processing” instead of an immediate result, which needs honest design: status pages, notifications and clear errors.
- Eventual consistency means different parts of the system can briefly disagree.
- Debugging is harder, because the work no longer happens inside one request. Use a correlation ID that follows the job from the request through the queue to the worker.
- Queues add infrastructure to run, monitor and pay for.
- Not everything belongs in a queue. Fast local work does not need one, and some results, such as a payment authorization decision, are needed immediately.
Tools to start with
1. Your broker’s dashboard, plus queue depth and oldest message age metrics
2. Distributed tracing with a correlation ID across request, queue and worker
3. A job dashboard for your library, such as Horizon for Laravel or Bull Board for BullMQ
4. Load tests that include worker crashes and duplicate messages
5. A “kill the worker mid job” test, run on purpose
A queue is a promise to do the work later. Make sure you can keep that promise twice.
