Side Effects of Micronaut Kafka Listener with Error strategy RETRY_ON_ERROR and batch=true

Samik Bandopadhyay

3 min read

·

Jan 11, 2026

--

Listen

Share

Introduction

Micronaut Kafka official documentation explicitly mentions that the error strategies apply only for non-batch message processing. But if the application is of high throughput and there is a real need of controlled replay of the records, the developers are forced to use error strategies with batch=true which is directly in contrast with what is mentioned in the documentation. We have implemented a similar use case in production. In this post, we are going to discuss the side effects of using ErrorStrategy RETRY_ON_ERROR with batch=true in production.

Our Use Case

We have a high throughput(100k events/sec) micronaut-kafka application which uses in memory caches for data look up. The caches are updated in real time. When the caches are being updated, the application should halt the processing and retry all the messages until the caches are up to date with the source. A specific custom exception is being thrown while caches are being synced which forces the error strategy to retry continuously until the syncing is completed. So, high throughput forces us to use batch=true while error strategy RETRY_ON_ERROR comes into picture to do controlled replay of the batches until cache syncing is done.

Listener Configuration

@Singleton
@KafkaListener(
     offsetReset = OffsetReset.EARLIEST,
     errorStrategy = ErrorStrategy(
         retryCount = Int.MAX_VALUE,
         handleAllExceptions = true,
         retryDelay = "1s",
         value = ErrorStrategyValue.RETRY_ON_ERROR,
     ),
     offsetStrategy = OffsetStrategy.DISABLED,
     batch = true,
     groupId = "\${listener.group-id}",
     threadsValue = "\${listener.threads:10}",
 )
 class EventConsumer(
       private val eventService : EventService
  ) {
     
     @Topic("topic1", "topic2", "topic3", "topic4")
     fun onEvent(consumerRecords: List<ConsumerRecord<String, String>>, acknowledgment: Acknowledgement) = runBlocking {
         consumerRecords
            .map { async { lookUPAndJoin(it) } }
            .awaitAll()
            //Do something
  
         acknowledgment.ack()
     }

private suspend fun lookUPAndJoin(record : ConsumerRecord<String, String>) = 
         record.value
               .toEventList() //deserialize
               .mapNotNull {
                   try {
                       eventService.lookUpAndJoin(it)
                   } catch(ex : RuntimeException) {
                       when(ex) {
                         is CacheNotReadyException -> throw ex
                         is KnownException -> //log it or do something
                         else -> throw ex                           
                       }
                      null
                     }
               }

Behaviour in Production

It behaves in production environment exactly the way we want it to behave. When the caches are getting synced, it keeps on throwing CacheNotReadyException and micronaut keeps on replaying the batch again and again due to RETRY_ON_ERROR ErrorStrategy until the cache syncing is done i.e. lookUpAndJoin() stopped throwing CacheNotReadyException. This behaviour is fully consistent with the configuration. The inconsistency arises when an unknown exception is thrown due to bad incoming data or some other reason for which there are no handlers within the code. Shortly after the prod deployment, we encountered one such incident and one of the on call engineers panicked and he terminated the pod where the replay was happening. And the issue got resolved which was never supposed to happen! Logs explicitly showed that the payload was failing due to the Schema Validation error and never supposed to resolve on its own. Basically, the entire batch of ConsumerRecords got dropped and we lost all those messages. The funny thing is that this behaviour is not consistent at all i.e. it doesn’t drop the batches every time we terminate the errored out pod. One thing we understood is that if we handle all the known exceptions related to application and data and the application can recover on its own without manual intervention like redeployment or termination of pods, this setup works just fine..

Constraints

Below are the scenarios when we can plan to have this combination:

• Application is of high throughput and there is a strong need to replay the batch for a scenario which will most likely be resolved within some stipulated time.
• All known exceptions are being handled and applications can auto. recover without manual intervention.

Below are the bottlenecks of this combination:

• This will not guarantee zero data loss. When an unhandled exception occurs, termination of pods leads to the possibility of dropping the entire batch of ConsumerRecords.
• This follows at-least-once semantic so the downstream may expect duplicates when RETRY happens.

Conclusion

Since this approach is not recommended, this should mostly be avoided. But if the constraints in the use case forces us to do so, we have to implement this in production with care. All the guardrails mentioed above should be in place.