Kafka Retry Strategy — Design Before Configuration
Retry strategy has become one of the most common design challenges in event-driven systems. Across microservice architectures and messaging patterns, development teams need to design and implement efficient retry strategies.
We spend a lot of time discussing configuration details such as maximum retry attempts, exponential backoff, multipliers, topic naming conventions, and retry delays, but surprisingly little time discussing the engineering decisions that define these configurations. This article shows how careful analysis and design can lead to a simpler, more robust, and more maintainable implementation.
Problem analysis
One of the most challenging questions we face when designing and implementing Kafka consumers is deciding how to handle failures. At the initial stage of designing failure handling it’s worth creating a high-level categorisation of the possible failures. The flow diagrams of the service can be a good starting point for identifying them.
We should also determine which failures are classified as non-retryable errors and which ones are temporary and might be eligible for retry. In the next sections we will refine these categories a bit further based on whether the error is message-specific or not.
Execution model
Tutorials usually demonstrate the consumption and processing pipeline with a single downstream system call. In real-world systems, after the message has been consumed and deserialised, the service usually executes several operations. The composition of the execution pipeline varies between consumer services, but it typically contains one or more of the following steps:
One of the most common misconceptions among developers is related to what is really retried. We often talk about the retry configuration of different downstream systems, which may give the impression that during a retry attempt only the failed step of the execution pipeline is retried.
When retry re-executes the entire message processing pipeline, all steps will be executed in the next attempt with the original payload, without any prior knowledge about the last attempt’s execution context. The unit of the retry is the entire pipeline and not an individual phase. This means all steps will be re-executed including the ones that were successful last time. The complexity of the retry strategy tends to increase with the number of steps.
Idempotency & side effects
Idempotency is one of the most important aspects of temporary failure handling we need to understand and analyse. The retry strategy should consider not just whether the next attempt can succeed, but also the potential side effects of the entire pipeline’s repeated execution with the same payload. The nature of each stage and the idempotency characteristics of the individual downstream systems will determine the result of a repeated execution.
The strategy should address the following questions where applicable for each downstream call:
- Does the downstream system provide idempotency?
- How is duplication detected and handled?
- What is the idempotency retention period, if any?
- If a downstream system doesn’t provide idempotency, can it tolerate duplicates?
- What is the cost of repeated calls?
Ideally, all downstream system calls should be idempotent, for example multiple executions of a REST PUT or an idempotent SQL Update will produce the same result. This is generally the cleanest and easiest solution. If the downstream call is not naturally idempotent but the operation accepts an idempotency key to detect duplication, the service design must handle the downstream system’s response to a duplicate request. Supporting this mechanism increases implementation complexity. If the operation is not idempotent and can’t be made idempotent, delayed compensation might be required, which is usually done as a post-processing step outside the scope of the service itself.
Design principles
After we have identified what can fail and examined the consequences of repeated execution, it’s time to define a framework for deciding how these failures will be handled.
Get Csaba Filipinyi’s stories in your inbox
Join Medium for free to get updates from this writer.
Based on what we established previously, the DLT and retry configuration should be the outcome of our design and not drive it. Before we start considering any implementation details such as retry topics, delays, and backoff, we have to answer three fundamental questions:
- Can the next attempt realistically succeed?
- Can the service continue processing subsequent messages, or should processing be suspended?
- Can processing be repeated safely, including operations that already succeeded?
Non-retryable vs retryable
The main difference between retryable and non-retryable failures is not exception type but whether time can realistically lead to a different outcome. A temporary network failure can be resolved after seconds or minutes, making another attempt worthwhile. An invalid payload will remain invalid after minutes or even weeks without manual modification, making the automated retry pointless.
There are some guidelines for different types of downstream systems such as REST, DB, streams, etc., but the fundamental question we always have to ask is the same: Is it realistic to expect a different result in the future?
Message level vs Service level
The second aspect that needs to be analysed is whether the issue is payload-specific or part of a wider problem. Whereas a missing mandatory field is message-specific, a downstream-system authentication error can be caused by a fundamental configuration issue which will persist when the next payload is processed. The design should answer the following questions for each error scenario:
- Will the issue affect the processing of the next payload?
- Is it acceptable to stop the service if the failure occurs?
- Is the failure a fundamental infrastructure issue which requires operational intervention such as an authentication issue?
- What is the consequence of suspending processing?
Failure categories
Based on these questions, the failures can fall into the following categories:
- Retryable failures: next attempt can succeed and the entire pipeline can be re-executed safely
- Non-retryable message-level failures: another attempt is not expected to succeed, or processing cannot safely be repeated, but the service can continue processing subsequent messages
- Service-level fatal failures: next attempt is not expected to succeed and the service should be stopped
Retryable failures
These are the failures that the service can recover from over time, and the entire pipeline can be re-executed safely including operations that previously succeeded. Existing failure categories for different types of downstream systems such as REST services can be used as guidance, but it’s worth considering how a particular scenario fits into our system.
Non-retryable message-level failures
A scenario falls into this category when:
- Another attempt is not expected to succeed or the pipeline can’t be re-executed safely including previously successful operations; as a result, another attempt could repeat side effects, waste resources, or delay the processing of subsequent payloads.
- The current message failed, but the issue is most likely isolated at the message level and the subsequent message processing is not expected to be affected.
From a design perspective, the objective is no longer to recover by replaying the message, but to remove the problematic payload from the flow and preserve enough information for offline analysis, operational support, and potential replay.
Retry exhaustion requires separate consideration. Its original categorisation remains valid, but once the retries are exhausted, its handling becomes equivalent to a non-retryable message-level failure.
Service-level fatal failures
Service-level fatal failures are treated here as non-retryable failures requiring intervention. Failures that are most likely symptoms of a wider problem fall into this category. As a result, the subsequent message processing is not expected to succeed due to the same root cause. The service design should provide a mechanism to prevent a large volume of occurrences of the same failure.
Conclusions
In this article, we reviewed the Kafka retry and error handling policy from a design perspective, and came to the following conclusions:
- We saw that the retry strategy is an integral part of the service design, and the more downstream calls we have in the execution pipeline, the more complex the retry design can be, and the more attention it requires.
- The retry settings have to be derived from the retry policy and not vice versa.
- Retrying a previously failed message will re-execute the entire pipeline, including all downstream system calls, not just the failed action.
- Retry design must consider the side effects of repeated execution, including the idempotency characteristics of the downstream systems.
- Retryable errors can realistically be resolved over time, making the payload eligible for retry.
- Non-retryable failures are not expected to become recoverable through automated repetition, or cannot safely be retried.
