Kafka Retry Strategies in Spring Boot: Blocking vs Non-Blocking, Retry Topics & DLT — Explained Simply

Anil Goyal

17 min read

·

Oct 26, 2026

--

1

Listen

Share

A step-by-step guide to building a reliable Kafka consumer in Spring Boot that retries failed messages safely — comparing blocking and non-blocking retries, using retry topics, and parking the hopeless ones in a Dead Letter Topic (DLT).

Why do we even need retries?

Imagine you run a service that reads orders from a Kafka topic and saves them to a database.

@KafkaListener(topics = "orders")
public void onOrder(Order order) {
    database.save(order);   // what if the database is down right now?
}

Most of the time this works. But sometimes:

• The database is temporarily down.
• A downstream REST API returns a 503 for a few seconds.
• A network blip causes a timeout.

These are transient failures — if you just wait a moment and try again, it will probably succeed. Throwing the message away would lose a real order. Blindly retrying forever would get you stuck. We need something smarter.

Goal: retry a failed message a few times with a delay between attempts, and if it still fails, set it aside safely so a human can look at it later — all without slowing down the healthy messages behind it.

The naive approach: blocking retries (and why it hurts)

The simplest retry is a loop inside the consumer:

@KafkaListener(topics = "orders")
public void onOrder(Order order) {
    for (int attempt = 1; attempt <= 5; attempt++) {
        try {
            database.save(order);
            return; // success
        } catch (Exception e) {
            sleep(5000); // wait 5 seconds and try again
        }
    }
    // give up...
}

This is called a blocking retry because the consumer thread is stuck sleeping and retrying this one message.

Why it hurts

A Kafka consumer reads messages from a partition in order, one at a time. While you sleep 5 seconds × 5 attempts = 25 seconds on one bad order, every other order behind it on that partition is frozen. One poisoned message can stall thousands of good ones. This is the problem non-blocking retries solve.

The big idea: non-blocking retries

Instead of sleeping on the main consumer, we move the failed message to a separate “retry topic” and immediately move on to the next message.

Spring Kafka can automatically create a chain of topics for you:

orders                 ← main topic (consumer never blocks here)
orders-retry-0         ← 1st retry, after a short delay
orders-retry-1         ← 2nd retry, longer delay
orders-retry-2         ← 3rd retry, even longer delay
orders-dlt             ← Dead Letter Topic (gave up — park it here)

Here is the mental model:

Each retry topic has its own consumer and its own delay. Because the delay happens on the retry topic — not the main topic — the main topic keeps flowing at full speed.

Press enter or click to view image in full size

That’s the whole idea. The rest of this blog explains how it works under the hood so you can configure it confidently.

Core concept: Kafka offsets and commits

To understand retries you must first understand offsets. This is the single most important concept, so let’s go slow.

Every message in a Kafka partition has a number called an offset — think of it as a line number in a file:

Partition 0:
  offset 100 → order-A
  offset 101 → order-B
  offset 102 → order-C

When your consumer finishes a message, it commits the offset. Committing means telling Kafka:

“I have successfully processed everything up to here. If I restart, start me from the next one.”

Consumer commits offset 100  →  "next time, give me 101"

Why this matters for retries

A commit is basically a bookmark. Here is the key rule:

Kafka does not have a “mark this one message as done” button. The committed offset is just a single moving bookmark that says “everything before me is finished.”

This creates a subtle trap. Suppose you process messages out of order:

offset 100 (order-A) → FAILS, you want to retry it later
offset 101 (order-B) → SUCCESS, you commit offset 101

By committing 101, you just told Kafka “everything up to 101 is done” — which silently includes the failed order-A at 100. If the app restarts, order-A is gone forever.

This is exactly why we can’t just “skip and keep going.” We need a safe way to set the failed message aside before the bookmark moves past it. That safe place is the retry topic.

Acknowledgment modes (and why RECORD is the sweet spot)

Spring Kafka lets you choose when the offset gets committed. This setting is called the ack mode (acknowledgment mode). The common options:

Press enter or click to view image in full size

For non-blocking retries, RECORD is the recommended mode. Let's see why by comparing it with MANUAL.

How RECORD works

spring.kafka.listener.ack-mode=record

With RECORD:

• Your listener returns normally → the container commits that record’s offset. ✅
• Your listener throws → the ack mode steps aside and the configured error handler takes over (it retries, and eventually commits). Exactly how depends on your retry model

The one idea to hold onto:

Ack mode = when to commit on success. Error handler = what to do on failure. Two separate concerns.

Either way you write zero acknowledgment code — the container and error handler handle it.

@KafkaListener(topics = "orders")
public void onOrder(Order order) {
    database.save(order);
    // return normally → container commits (that's RECORD).
    // throw → the error handler takes over 
}

Why RECORD and not MANUAL?

With MANUAL ack mode you take responsibility for calling acknowledge():

@KafkaListener(topics = "orders")
public void onOrder(Order order, Acknowledgment ack) {
    database.save(order);
    ack.acknowledge(); // you commit explicitly
}

The official Spring Kafka guidance for non-blocking retries is clear:

“You can set the AckMode mode you prefer, but RECORD is suggested.” — Spring Kafka reference: How the Pattern Works

Important: even with non-blocking retries, the framework manages the offset commit for you on the failure path — regardless of ack mode. When your listener throws, you simply let the exception propagate; the retry infrastructure forwards the record to the next retry topic (or DLT) and commits. You do not hand-write “ack in the catch and rethrow" logic.

So why prefer RECORD?

• With RECORD you write no acknowledgment code at all. On success the container commits; on failure the framework forwards + commits. Nothing to get wrong.
• MANUAL is allowed, but it's a configuration concern, not just adding ack.acknowledge(). Per the docs, a manual AckMode with non-blocking retries requires asyncAcks=true and the error handler's seekAfterError=false (set automatically since 2.9.10 / 3.0.8; older versions needed manual configuration, and the default logging DLT handler wasn't compatible with manual acks at all). That's extra setup for no real benefit here.

A Kafka fact worth knowing (for either mode): not committing does not mean the next poll() replays the same message. Within a running consumer, Kafka tracks your position in memory, so each poll returns the next record regardless of commits. The committed offset only matters on a rebalance/restart (a new consumer resumes from the last committed offset) or when the error handler performs an explicit seek(). So a missed commit doesn't cause an instant re-poll loop — it causes reprocessing after a restart/rebalance.

Rule of thumb: With non-blocking retries, prefer RECORD — it's the Spring-suggested mode, needs zero ack code, and the framework handles both the retry-topic handoff and the offset commit (in the correct order) for you. Reach for MANUAL only if you have a specific reason, and then configure asyncAcks + seekAfterError=false as the docs require.

Retryable vs non-retryable errors

Not every failure deserves a retry. Retrying only makes sense for transient problems.

Press enter or click to view image in full size

Retrying a permanent error is pointless: you’d burn through every attempt and still fail, just slower. For those, we want to skip the retries and go straight to the DLT.

We express this with custom exceptions:

// Transient → worth retrying
public class TransientProcessingException extends RuntimeException {
    public TransientProcessingException(String msg, Throwable cause) { super(msg, cause); }
}

// Permanent → do NOT retry, send straight to DLT
public class NonRetryableException extends RuntimeException {
    public NonRetryableException(String msg) {

Then we tell Spring Kafka which exceptions should not be retried. The flow looks like this:

Press enter or click to view image in full size

Putting it together: a working Spring Boot example

Now let’s build the whole thing. We’ll use Spring Boot + spring-kafka. This is generic code you can drop into any project.

7.1 Dependencies (build.gradle)

dependencies {
    implementation 'org.springframework.boot:spring-boot-starter'
    implementation 'org.springframework.kafka:spring-kafka'
}

7.2 Configuration (application.properties)

# --- Connection ---
spring.kafka.bootstrap-servers=localhost:9092
spring.kafka.consumer.group-id=order-consumer-group

# --- Start from the beginning the first time, and let the container manage commits ---
spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.enable-auto-commit=false# --- THE IMPORTANT ONE: commit each record after it succeeds ---
spring.kafka.listener.ack-mode=record

enable-auto-commit=false hands commit control to the Spring container (instead of Kafka committing on a timer). Combined with ack-mode=record, the container commits precisely after each successful record. This pairing is what makes the retry logic reliable.

7.3 The custom exceptions

public class TransientProcessingException extends RuntimeException {
    public TransientProcessingException(String message, Throwable cause) {
        super(message, cause);
    }
}

public class NonRetryableException extends RuntimeException {
    public NonRetryableException(String message) {
        super(message);
    }
}

7.4 The retry topic configuration

This single bean wires up the whole retry-topic + DLT machinery.

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.RetryTopicConfiguration;
import org.springframework.kafka.config.RetryTopicConfigurationBuilder;
import org.springframework.kafka.core.KafkaTemplate;

@Configuration
public class RetryTopicConfig {    @Bean
    public RetryTopicConfiguration retryTopicConfiguration(KafkaTemplate<String, Object> template) {
        return RetryTopicConfigurationBuilder
                .newInstance()
                // Retry up to 3 times (so: 1 original attempt + 3 retries)
                .maxAttempts(4)
                // Wait 5s, then 10s, then 20s between attempts (exponential backoff)
                .exponentialBackoff(5000, 2, 60000)
                // These exceptions skip the retries and go STRAIGHT to the DLT
                .notRetryOn(NonRetryableException.class)
                .create(template);
    }
}

Let’s decode the key settings:

• maxAttempts(4) → 1 initial try + 3 retries.
• exponentialBackoff(5000, 2, 60000) → first delay 5000 ms, each next delay ×2, capped at 60000 ms. So delays are 5s → 10s → 20s.
• notRetryOn(NonRetryableException.class) → if the listener throws this, skip all retry topics and publish directly to the DLT.
• Everything else (DB timeouts, etc.) is retried.
• By default Spring Kafka creates one retry topic per attempt (orders-retry-0/1/2). If several attempts share the same delay, you can collapse them onto a single shared retry topic with .useSingleTopicForSameIntervals().

7.5 The listener

import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.stereotype.Component;

@Component
public class OrderListener {    private final OrderService orderService;    public OrderListener(OrderService orderService) {
        this.orderService = orderService;
    }    @KafkaListener(topics = "orders", groupId = "order-consumer-group")
    public void onOrder(Order order) {
        orderService.process(order);
        // return normally  → container commits the offset (RECORD mode)
        // throw an exception → retry infrastructure takes over
    }
}

7.6 The service that decides retryable vs non-retryable

import org.springframework.stereotype.Service;

@Service
public class OrderService {    private final Database database;    public OrderService(Database database) {
        this.database = database;
    }    public void process(Order order) {
        // Permanent problem → do NOT retry
        if (order.getCustomerId() == null) {
            throw new NonRetryableException(
                "Order is missing customerId; retrying will never help");
        }        try {
            database.save(order);
        } catch (DatabaseUnavailableException e) {
            // Transient problem → worth retrying
            throw new TransientProcessingException(
                "Database temporarily unavailable, will retry", e);
        }
    }
}

That’s the entire setup. No manual error handlers, no retry loops, no Thread.sleep. Spring Kafka creates the retry topics and the DLT, schedules the delays, and routes messages for you.

Walking through the full message flow

Let’s trace a single order through the system. Assume our config: 4 attempts total, delays 5s → 10s → 20s, DLT at the end.

Case A: Transient failure that eventually succeeds

1. Consumer reads order-A from "orders" (offset 100).
2. database.save() throws DatabaseUnavailableException.
3. Listener rethrows TransientProcessingException.
4. The error handler PUBLISHES order-A to "orders-retry-0"
   and COMMITS offset 100 on the main topic.
      → The main consumer's job for order-A is DONE. It moves to order-B immediately.
5. After 5s, the retry-0 consumer reads order-A and tries again.
      → Still failing? Publish to "orders-retry-1", commit retry-0's offset.
6. After 10s, retry-1 consumer tries again.
      → Succeeds this time! Commit retry-1's offset. order-A is fully processed. ✅

The crucial insight from step 4: the main consumer commits the offset and moves on the instant it hands the message to the retry topic. It does not wait. This is why the main topic never blocks.

Press enter or click to view image in full size

Case B: It fails every single time

orders        → fail → publish to retry-0, commit
retry-0 (5s)  → fail → publish to retry-1, commit
retry-1 (10s) → fail → publish to retry-2, commit
retry-2 (20s) → fail → publish to orders-dlt, commit
orders-dlt    → message parked here for a human / alerting system

The message is never lost — it ends up safely in the DLT. You can monitor the DLT, alert on it, and even replay messages back to the main topic once the underlying bug is fixed.

Case C: Non-retryable (bad data)

orders → order with no customerId → NonRetryableException
       → error handler sees it's in notRetryOn list
       → publish STRAIGHT to orders-dlt, commit
       (retry-0/1/2 are skipped entirely — no point waiting)

Why committing after a failure is safe

A natural worry: “If the main consumer commits offset 100 right after a failure, isn’t order-A at risk before any retry succeeds?”

No. The commit happens only after the record was already published to orders-retry-0 — a durable Kafka topic on disk. The record isn't dropped; its responsibility is handed off. Each topic in the chain (main, each retry, the DLT) keeps its own offset, and every stage commits only once the record is safely on the next topic — so there's never an "uncommitted" record floating around. And because each retry topic has its own consumer thread, the main partition never waits: it committed and moved on long ago.

Mental model: a relay race. Each runner (topic/consumer) only lets go of the baton (commits its offset) after the next runner has firmly grabbed it (the record is safely on the next topic). The baton is never dropped mid-handoff.

The other model: blocking retries with DefaultErrorHandler

Everything above used non-blocking retries (separate retry topics). But Spring Kafka also supports the older, simpler model: blocking retries using a DefaultErrorHandler. It's worth knowing because many existing services use it, and it's the right choice when strict ordering matters more than throughput.

The key difference

In the blocking model, no retry topics are created. The record is retried in place, from the same original topic, by re-invoking the same listener method on the same consumer thread — which means the partition is blocked during the retries.

Breaking that down:

• No retry topics are created. There is no orders-retry-0/1/2 chain — the only extra topic involved is the DLT (and only once retries are exhausted).
• Retried in place, from the same original topic. The record is never republished somewhere else for retrying; it stays on its original topic/partition.
• The same @KafkaListener method is re-invoked. Spring Kafka simply calls your listener again with the same record — your business logic runs from the top each attempt.
• On the same consumer thread. The thread that polled the record is the one doing the waiting and retrying.
• The partition is blocked. Because that thread is busy retrying one record, every message behind it on the same partition waits (ordering is preserved, throughput is sacrificed).

Non-blocking (retry topics):  fail → move to orders-retry-0 → main topic keeps flowing
Blocking (DefaultErrorHandler): fail → retry same record in place → partition waits

Press enter or click to view image in full size

How it works, step by step

1. The listener throws an exception.
2. The DefaultErrorHandler catches it and waits for the backoff delay.
3. It then re-delivers the same record to the same @KafkaListener method (same thread, same topic — nothing is published anywhere).
4. Repeat until it succeeds or the max attempts are used up.
5. When attempts are exhausted, the recoverer (DeadLetterPublishingRecoverer) publishes the record to the DLT.
6. After the DLT publish succeeds, Spring Kafka commits the offset and the consumer moves on.

During steps 2–4, every other message behind this one on the same partition is waiting — that’s the “blocking” part.

The code

You don’t use RetryTopicConfiguration here. Instead you register a single DefaultErrorHandler bean and attach it to the listener container factory:

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.core.KafkaTemplate;
import org.springframework.kafka.listener.DeadLetterPublishingRecoverer;
import org.springframework.kafka.listener.DefaultErrorHandler;
import org.springframework.util.backoff.FixedBackOff;

@Configuration
public class BlockingRetryConfig {

    @Bean
    public DefaultErrorHandler errorHandler(KafkaTemplate<String, Object> template) {

        // Recoverer: where a record goes after all retries are exhausted.
        // By default it publishes to "<topic>-dlt" (here: "orders-dlt").
        DeadLetterPublishingRecoverer recoverer = new DeadLetterPublishingRecoverer(template,
                (record, exception) -> new org.apache.kafka.common.TopicPartition(
                        record.topic() + "-dlt", record.partition()));

        // Retry settings: wait 5 seconds between attempts, retry 3 times.
        // FixedBackOff(interval, maxRetries) → 3 retries = 4 total attempts.
        FixedBackOff backOff = new FixedBackOff(5000L, 3L);

        DefaultErrorHandler handler = new DefaultErrorHandler(recoverer, backOff);

        // Permanent errors: skip retries, send straight to the DLT.
        handler.addNotRetryableExceptions(NonRetryableException.class);

        return handler;
    }
}

Then tell your listener container factory to use it:

import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.kafka.config.ConcurrentKafkaListenerContainerFactory;
import org.springframework.kafka.core.ConsumerFactory;
import org.springframework.kafka.listener.DefaultErrorHandler;

@Configuration
public class KafkaConsumerConfig {

    @Bean
    public ConcurrentKafkaListenerContainerFactory<String, Object> kafkaListenerContainerFactory(
            ConsumerFactory<String, Object> consumerFactory,
            DefaultErrorHandler errorHandler) {

        ConcurrentKafkaListenerContainerFactory<String, Object> factory =
                new ConcurrentKafkaListenerContainerFactory<>();
        factory.setConsumerFactory(consumerFactory);

        // Attach the blocking error handler.
        factory.setCommonErrorHandler(errorHandler);

        return factory;
    }
}

The listener itself stays exactly the same — it has no idea retries are happening:

@KafkaListener(topics = "orders", groupId = "order-consumer-group")
public void onOrder(Order order) {
    orderService.process(order);
    // throw → DefaultErrorHandler retries THIS record in place, same method, same thread
}

What to remember

Press enter or click to view image in full size

Note: Don’t configure both a RetryTopicConfiguration and a custom DefaultErrorHandler for the same listener expecting them to combine arbitrarily — the non-blocking retry infrastructure installs its own error handler. Pick the model that fits your ordering vs. throughput needs. (Advanced setups can layer short blocking retries inside non-blocking retries, but keep it simple unless you truly need it.)

The four combinations: ack mode × retry model

You’ve now met two independent choices:

• Ack mode — who commits the offset on success? RECORD (framework) or MANUAL (you).
• Retry model — what happens on failure? Blocking (DefaultErrorHandler) or non-blocking (retry topics).

Combine them and you get four setups. Here’s each one as a simple flow so you can see exactly who does what.

Press enter or click to view image in full size

1. RECORD + blocking

The simplest safe setup when order matters. You write no ack code.

success → framework commits → next record
failure → DefaultErrorHandler retries THE SAME record in place (partition waits)
             → a retry succeeds? → framework commits → next record
             → all retries fail? → publish to DLT → framework commits → next record

• You write: business logic only.
• Partition: blocked during retries (ordering preserved).
• Commits: 100% framework.

2. RECORD + non-blocking

The recommended high-throughput setup (what most of this blog uses). You write no ack code.

success → framework commits → next record
failure → framework publishes to orders-retry-0 → THEN commits → next record (no waiting)
             (a separate retry-topic consumer thread then processes it)
                → success? → framework commits
                → fail?    → publish to next retry topic → commit → ... → DLT

• You write: business logic only.
• Partition: never blocked (throughput preserved, ordering lost).
• Commits: 100% framework, always in the safe order (publish first, then commit).

MANUAL + blocking

Uncommon — you opt into manual acks but still retry in place.

success → YOU call ack.acknowledge() → next record
failure → you throw → DefaultErrorHandler retries THE SAME record in place (partition waits)
             → a retry succeeds? → your listener runs from the top → YOUR ack runs
             → all retries fail? → publish to DLT → FRAMEWORK commits (your code never runs)

• You write: ack.acknowledge() on the success path only.
• Partition: blocked during retries (ordering preserved).
• Commits: you on success; framework on DLT recovery.

4. MANUAL + non-blocking

Allowed, but mostly a configuration exercise — you do not hand-write failure acks.

success → YOU call ack.acknowledge() → next record
failure → you let the exception propagate
             → FRAMEWORK publishes to orders-retry-0 and commits → next record
             (retry-topic consumer repeats the same pattern → ... → DLT)

• You write: ack.acknowledge() on the success path only (+ config asyncAcks=true, seekAfterError=false).
• Partition: never blocked (throughput preserved, ordering lost).
• Commits: you on success; framework on every failure/handoff/DLT.

How to choose

Bottom line: unless you have a specific reason to manage commits yourself, use RECORD and pick blocking (order) or non-blocking (throughput) based on your needs. The two RECORD combinations cover the vast majority of real services.

Press enter or click to view image in full size

Who creates the retry topics and how are they named?

You don’t create them manually. Spring Kafka’s retry infrastructure creates them automatically at startup (if your broker allows auto topic creation, or via a NewTopic bean / your platform's topic provisioning).

Default naming is the main topic name + a suffix:

orders            ← main
orders-retry-0    ← retry attempt 1
orders-retry-1    ← retry attempt 2
orders-retry-2    ← retry attempt 3
orders-dlt        ← dead letter topic

• The number of retry topics = number of retries (here, 3).
• Each retry topic has its own partitions and its own consumer group offset.
• You can customize the suffixes, partition count, and even use a single shared retry topic instead of one-per-attempt (useSingleTopicForSameIntervals() when delays are equal).

If your organization provisions topics centrally (common in regulated / platform-managed environments), make sure the retry and DLT topics are included in that provisioning, or enable auto-creation for them.

Common pitfalls

1. Reaching for MANUAL ack with non-blocking retries. You don't need it — the framework commits for you on both success and failure, and the docs suggest RECORD. If you do use a manual AckMode, it's a configuration task (asyncAcks=true + the error handler's seekAfterError=false), not a "ack in success and failure" coding pattern. For most cases, just use RECORD and write no ack code.
2. Mixing the two retry models. The retry-topic (non-blocking) infrastructure installs its own error handler. If you also register a custom DefaultErrorHandler on the container factory for the same listener, you may override or conflict with the retry/DLT behavior. Pick one model: RetryTopicConfiguration for non-blocking, or DefaultErrorHandler for blocking.
3. Retrying permanent errors. Malformed data will fail every time. Add those exception types to notRetryOn(...) so they jump straight to the DLT instead of wasting all your attempts.
4. Forgetting to monitor the DLT. Messages in the DLT are stuck. Without alerting, they silently pile up. Always monitor DLT lag/size.
5. Making the producer “fire and forget” when you need delivery guarantees. If publishing the result downstream can fail and you want that retried too, make the publish synchronous (wait for the broker ack) so a failure actually throws and triggers the retry path. Async publishing returns immediately and a failure may go unnoticed — a reasonable trade-off only if losing that publish is acceptable.

Final takeaway

Non-blocking retries give you the best of both worlds: resilience (transient failures are retried with sensible delays) and throughput (one bad message never stalls the healthy ones). The recipe is small:

1. Use ack-mode=record so offsets commit exactly on success.
2. Define a RetryTopicConfiguration with your attempts, backoff, and notRetryOn list.
3. Throw a retryable exception for transient failures and a non-retryable one for permanent failures.

Let Spring Kafka handle the topics, delays, routing, and the DLT. Your listener stays clean, and your pipeline stays fast and reliable.