I Passed the Kafka Interview. Then the Staff Engineer Asked Me to Delete One Line
I knew partitions, offsets, consumer groups, and exactly-once semantics. One innocent-looking retry turned a perfectly healthy event pipeline into a system that could ship canceled orders.
The interviewer gave me a Java class with seventy lines of code and asked whether I would approve it for production. The service consumed Kafka events, updated order state in PostgreSQL, and committed offsets after successful processing. There were retries, structured logs, metrics, and enough annotations to make the implementation look like somebody had taken a Spring Boot certification exam during a nervous breakdown.
I spent the first ten minutes explaining Kafka’s ordering guarantees. Records within the same partition are ordered, consumers in the same group divide partition ownership, and committing offsets after database work reduces the risk of losing an operation. The interviewer agreed with everything I said, which should have made me suspicious.
Then he pointed at one line.
retryTemplate.execute(context -> process(event));He asked me to remove it.
I laughed because I assumed he was testing whether I understood transient failures. Removing retries from a production consumer sounded irresponsible, especially when database deadlocks, network interruptions, and temporary downstream failures were part of the normal operating environment. He asked whether I knew what happened to later events for the same order while that retry was running.
I told him Kafka preserved ordering within a partition. He said that was true, then asked whether our application preserved the same ordering after moving failed records into a separate retry topic.
That was when the interview stopped being comfortable.
The following story is a fictional composite of realistic Senior and Staff Backend Engineer interview discussions. The code and failure mechanisms are based on actual Kafka, PostgreSQL, and Spring Boot behavior, while the interview conversation and outcome are illustrative.
The System That Looked Correct on Paper
The company operated an order-processing platform that handled several million events per day. Orders moved through a sequence of business states, including creation, payment confirmation, fulfillment, cancellation, and shipment. Kafka distributed these events between services, while PostgreSQL stored the authoritative order state.
The event stream used orderId as the Kafka message key. That meant events for the same order were normally assigned to the same partition, assuming a stable partitioning strategy and partition count. Within a partition, Kafka preserved the order in which records were appended.
The consumer processed events and updated the database. When processing failed, the application published the record to a retry topic so the original consumer could continue making progress. After several unsuccessful attempts, the event moved to a dead-letter topic for investigation.
The implementation appeared reasonable:
@KafkaListener(topics = "order-events")
public void consume(OrderEvent event) {
try {
orderService.apply(event);
} catch (TransientException exception) {
retryPublisher.publish(event);
}
}The interviewer asked what happened when an ORDER_PAID event failed because PostgreSQL was temporarily unavailable. The consumer would publish it to the retry topic and continue processing the original partition. That behavior prevented one failed event from blocking unrelated orders.
He then asked what happened when an ORDER_CANCELLED event for the same order arrived immediately afterward.
I explained that the cancellation event would be processed normally. The interviewer asked whether the payment event had already been applied. It hadn’t, because it was waiting in the retry topic.
The original partition had preserved its record order. The application had broken the business ordering by moving one event into a separate processing path.
The interviewer asked whether that mattered.
I said it depended on the order state machine. He smiled and asked me to define the state machine before suggesting another retry configuration.
The First Failure: Kafka Ordering Didn’t Survive the Retry Topic
We began with an order that had been created but not yet paid. The payment service published an ORDER_PAID event, followed shortly by an ORDER_CANCELLED event triggered by a customer support operation. Both events used the same Kafka key and were appended to the same original partition.
The payment event failed during database processing and moved to the retry topic. The cancellation event continued through the main consumer and updated the order state to CANCELLED. Several seconds later, the payment event returned from the retry path.
The interviewer asked what the consumer should do with the delayed payment event. My first answer was that it should apply the event because Kafka had delivered it successfully and the database operation had previously failed. He asked whether a canceled order should become paid again.
I reconsidered. The correct behavior depended on the business transition rules, and a delayed event couldn’t simply overwrite the current state. The application needed to determine whether the transition remained valid in the context of the order’s authoritative state.
The interviewer asked whether we could fix the problem by increasing retry delays. I explained that a longer delay might reduce pressure on an unhealthy dependency, but it would make reordering more likely rather than eliminate it. A shorter delay could still allow later events to overtake the failed event.
He then asked whether Kafka’s exactly-once processing guarantees would solve the problem. I explained that Kafka transactions could coordinate certain Kafka read-process-write workflows, but they didn’t automatically preserve application-level ordering after events were deliberately moved between independent topics. They also didn’t make arbitrary PostgreSQL updates atomic with Kafka transactions.
The interviewer wanted me to distinguish broker ordering from business ordering. Kafka could preserve append order within a partition, while the application could still violate state transition rules by processing related events through independent retry paths. The system needed an explicit strategy for maintaining or safely relaxing those ordering requirements.
I had understood Kafka’s guarantee correctly. I had misunderstood where that guarantee stopped.
The Second Failure: The Database Accepted an Impossible Transition
The interviewer opened the order service implementation. The method loaded an order, changed its status, and saved it through a Spring Data repository. The code used @Transactional, and the developer had assumed that the annotation was sufficient to keep order state consistent.
The relevant logic looked like this:
@Transactional
public void apply(OrderEvent event) {
Order order = repository.findById(
event.orderId()
).orElseThrow();
order.setStatus(event.targetStatus());
repository.save(order);
}The interviewer asked what happened if a delayed ORDER_PAID event arrived after the order had already been canceled. I explained that the method would load the canceled order, overwrite its status with PAID, and commit the transaction. The database would preserve the update correctly, even though the business transition was invalid.
He asked whether increasing the transaction isolation level would solve the problem. I said it wouldn’t automatically establish the allowed state transitions. Isolation could affect how concurrent transactions interacted, but the application still needed rules describing which transitions were valid.
The interviewer then introduced concurrent processing. Two events for the same order could be handled by different consumers if the application had routed them through separate topics or processing paths. Both transactions might read the same initial state and attempt conflicting updates.
I proposed optimistic locking using a version column. The interviewer asked whether a version conflict would tell us which business event should win. I explained that optimistic locking could detect a conflicting concurrent update, but it wouldn’t independently determine the correct transition order.
We discussed using event sequence numbers or aggregate versions assigned by the authoritative event producer. Each event would carry information that allowed the consumer to determine whether it was the next expected transition, a duplicate, or an out-of-order event. The application could then reject invalid transitions or defer events until missing predecessors were resolved.
The interviewer asked whether a timestamp would be enough. I explained that timestamps could be useful for diagnostics, but they weren’t necessarily a reliable ordering authority across distributed producers. Clock differences, delayed delivery, and concurrent event creation could make timestamp ordering ambiguous.
He asked how I would enforce the transition rules in PostgreSQL. I proposed conditional updates that checked the current state or expected version before applying a change. The update would succeed only when the stored state matched the transition’s preconditions.
A simplified example looked like this:
UPDATE orders
SET status = 'PAID',
version = version + 1
WHERE id = :order_id
AND status = 'CREATED'
AND version = :expected_version;The interviewer asked what a zero-row update meant. I explained that it could indicate a stale event, an invalid transition, or a concurrent modification, depending on the context. The application needed to inspect the durable state and classify the outcome rather than blindly retrying the same update.
That was the point where the discussion moved beyond Kafka configuration. The system’s correctness depended on a state machine enforced through durable state transitions. Kafka could deliver events in useful order, but the database needed to reject operations that violated the business model.
The Third Failure: The Retry Mechanism Created a Duplicate Shipment
The interviewer introduced another event type: SHIP_ORDER. The consumer received a shipment request, called an external fulfillment provider, and recorded the shipment reference in PostgreSQL. The implementation retried failed operations using the same general retry infrastructure.
He asked what happened if the fulfillment provider accepted the shipment request but the application timed out before receiving the response. I explained that the local service would classify the attempt as uncertain because the provider might have created the shipment. The interviewer asked what the retry worker would do.
The existing implementation would send the same request again.
He asked whether that could create two shipments. I said it could if the provider didn’t support idempotent requests or if the application failed to reuse a stable operation identity. The database transaction couldn’t reverse a shipment that had already been created in an external system.
The interviewer asked whether storing the Kafka event ID would prevent the duplicate shipment. I explained that it could help deduplicate local processing, but it wouldn’t necessarily prevent repeated external effects after a crash. The application needed to coordinate durable operation state with the provider’s idempotency or lookup capabilities.
We discussed a durable shipment operation record. Before invoking the provider, the application would persist a stable shipment identity and processing state. The provider call would use that identity where supported, and recovery would reconcile uncertain outcomes before initiating another independent shipment.
The interviewer asked what happened if the provider didn’t support idempotency keys. I explained that the system couldn’t guarantee duplicate prevention solely through local database constraints if it had no reliable way to identify or query an earlier external operation. The business would need a reconciliation process and potentially manual handling for uncertain outcomes.
He then asked whether the retry topic should contain shipment commands at all. I said it could, but only if the processing protocol made repeated attempts safe. The retry infrastructure shouldn’t treat every failed event as equivalent because a database read, an internal state update, and an external shipment request have different failure consequences.
The interviewer had started with one line of retry code. We were now discussing external side effects, durable operation identity, and the limits of exactly-once processing. The retry mechanism was no longer a simple resilience feature. It was part of the system’s correctness model.
The Fourth Failure: Our Dead-Letter Queue Hid a Business Incident
The interviewer moved to the dead-letter topic. The application sent events there after several unsuccessful attempts, and operators reviewed them through an internal dashboard. The original design treated the dead-letter queue as a way to keep the main consumer moving.
He asked what happened when an ORDER_PAID event entered the dead-letter topic while later events for the same order continued processing. I explained that the system could move forward without applying a required predecessor. The dead-letter queue might prevent consumer blockage while creating inconsistent business state.
He asked whether the application should stop processing every order when one event failed. I said that would be unnecessarily disruptive if the failure affected only one order. The system needed a way to isolate the affected aggregate while allowing unrelated orders to continue.
Get Production Systems’s stories in your inbox
Join Medium for free to get updates from this writer.
We discussed per-order processing state. If an event failed and later events depended on it, the application could mark that order as blocked or awaiting reconciliation. Other orders could continue processing, while events for the affected order would be deferred until the missing transition was resolved.
The interviewer asked whether Kafka partitions were sufficient for that isolation. I explained that a partition contained events for many orders, so pausing the entire partition could delay unrelated work. The application might need durable per-aggregate sequencing or a different processing model to isolate failures without sacrificing correctness.
He then asked how operators would know whether the dead-letter queue represented a technical problem or a business incident. I proposed metrics and alerts that distinguished malformed events, transient infrastructure failures, invalid state transitions, and unresolved external operations. A growing dead-letter topic wasn’t merely a backlog metric if it contained events required to complete customer orders.
The interviewer asked what information the dead-letter record should preserve. I suggested the original event identity, aggregate identity, payload, failure classification, attempt history, and relevant diagnostic context. The system also needed a controlled replay process that respected the current business state rather than blindly republishing old events.
He asked whether replaying a dead-letter event should bypass the state machine. I said it shouldn’t, because an old event could become invalid after later business changes. Replay needed to follow the same correctness rules as ordinary processing, with explicit operator decisions for exceptional recovery cases.
The discussion had moved into operational ownership. A dead-letter queue was useful for isolating failures, but it didn’t automatically resolve their consequences. The organization needed to understand which business workflows remained incomplete and how those workflows would be recovered safely.
The Architecture I Finally Proposed
With fifteen minutes remaining, the interviewer asked me to redesign the processing path. He wanted a system that could handle transient failures without silently violating order transitions. He also wanted unrelated orders to continue processing when one order encountered a problem.
I began by defining the aggregate boundary. Events for the same order needed a stable identity and an explicit sequence or version assigned by an authoritative component. The consumer would use that information to determine whether an event was expected, duplicated, or out of order.
The database would store the current order state and version. Applying an event would require a valid transition from the current state and the expected version. The update would be atomic, so concurrent attempts couldn’t independently overwrite the same state without detection.
For transient failures, I proposed distinguishing operations that could safely retry from those that required reconciliation. A temporary database failure could justify a bounded retry, but an external shipment timeout needed a durable operation identity and a way to determine whether the provider had already accepted the request. The retry policy would reflect the operation’s actual semantics.
The interviewer asked how I would avoid blocking unrelated orders. I explained that the system could isolate failed aggregates through durable per-order processing state, while allowing other orders to continue. The exact implementation would depend on throughput, ordering requirements, and operational complexity.
He asked whether I would still use retry topics. I said they remained useful for workloads where reordering was acceptable or where the application enforced sequencing independently. For strictly ordered aggregate transitions, moving events between topics required additional coordination to prevent later events from overtaking required predecessors.
The interviewer asked whether Kafka exactly-once semantics would be part of the design. I explained that Kafka transactions could be valuable for appropriate Kafka-to-Kafka processing workflows, but PostgreSQL state transitions and external fulfillment effects required their own correctness mechanisms. I wouldn’t describe the entire system as exactly once without defining the specific operation and guarantees.
We discussed testing next. I proposed integration tests that deliberately reordered events, duplicated messages, interrupted consumers after database commits, and simulated external provider timeouts. The tests would verify that the order state machine rejected invalid transitions and that recovery didn’t create duplicate shipments.
The interviewer asked how I would observe failures in production. I suggested tracking blocked aggregates, sequence gaps, invalid transitions, retry age, dead-letter classifications, and reconciliation backlog. Consumer lag remained useful, but it wasn’t enough to establish whether customer orders were progressing correctly.
The interviewer looked at the original retry line again. He asked whether I would delete it now.
I explained that I would remove the generic retry behavior from that processing path and replace it with failure handling appropriate to each operation. Some retries would remain, but they would no longer be allowed to bypass the ordering and business correctness rules.
He closed the code review and said we had covered enough.
Why This Interview Was Harder Than a Kafka Certification Exam
The interview didn’t require obscure Kafka configuration knowledge. It required understanding how broker guarantees interacted with application state, database transactions, and external side effects. I knew that Kafka preserved ordering within a partition, but I hadn’t initially considered how retry topics could invalidate the application’s assumptions.
I also understood idempotency in the abstract. The difficult part was applying it to operations that crossed independent systems, especially when the application couldn’t determine whether an external request had succeeded. A local deduplication record was useful, but it wasn’t a complete answer for repeated shipment requests.
The state machine discussion exposed another weakness. I had assumed that processing events in order would naturally produce valid state transitions. The interviewer showed that delayed, duplicated, or concurrently processed events required explicit validation against durable state.
The broader lesson was that infrastructure guarantees don’t replace domain correctness. Kafka can provide ordered records within partitions, PostgreSQL can enforce atomic updates, and Spring Boot can manage local transactions. The application must still define which transitions are allowed and how uncertain operations are recovered.
The Kafka Interview Preparation Framework I Would Use Today
I would begin by identifying the business aggregate and its state transitions. An order, payment, or shipment has rules that determine which operations are valid and how they relate to one another. Those rules should be explicit before choosing retry topics, consumer concurrency, or offset management strategies.
Next, I would define the ordering requirements. Kafka preserves order within a partition, but applications can introduce reordering through retries, asynchronous processing, multiple topics, or independent consumers. The design needs to establish whether reordering is acceptable and how invalid sequences are detected.
I would then examine idempotency and durable state. Duplicate delivery is a normal possibility in many processing workflows, so the application needs stable operation identities and appropriate database constraints. External side effects require additional care because local transactions can’t automatically control remote systems.
Failure recovery deserves its own design. A retry topic can improve progress, but it may also allow dependent events to overtake failed predecessors. Dead-letter queues isolate failures, yet the organization still needs a way to classify and recover incomplete business operations.
Finally, I would test the failure windows that matter. Duplicate messages, out-of-order events, consumer crashes, rebalance interruptions, and uncertain external outcomes should be part of the test strategy. The strongest tests verify business invariants rather than merely checking that the consumer eventually acknowledges records.
The Senior Backend Engineer Interview OS covers advanced backend coding and distributed systems scenarios. Engineers preparing for broader architecture interviews can also use the Senior & Staff System Design Interview Masteryfor deeper preparation around consistency, messaging, and failure recovery.
Frequently Asked Questions About Kafka Ordering and Senior Backend Interviews
Does Kafka guarantee message ordering?
Kafka preserves record order within an individual partition. It doesn’t provide a single global ordering across all partitions. Applications can also introduce reordering through retry topics, asynchronous processing, and other independent execution paths.
Can Kafka retry topics break message ordering?
Yes. Moving a failed record to a separate retry topic can allow later records from the original topic to be processed first. That may be acceptable for independent events but dangerous when later operations depend on earlier state transitions. The application needs an explicit ordering strategy.
How do I preserve order for events belonging to the same entity?
Use a stable partitioning key where appropriate and define how the application handles retries and failures. For stronger guarantees across independent processing paths, durable sequence numbers or aggregate versions can help detect missing or out-of-order events. The correct approach depends on the business workflow.
Does Kafka exactly-once semantics prevent duplicate database updates?
Not automatically. Kafka transactions provide guarantees for supported Kafka processing workflows, but they don’t make arbitrary PostgreSQL transactions atomic with Kafka operations. Applications need database-level correctness mechanisms for external state changes.
What is the difference between idempotency and ordering?
Idempotency ensures that repeating a logical operation doesn’t create additional unintended effects. Ordering determines the sequence in which operations are applied. A system can be idempotent while still applying valid operations in the wrong order, so both concerns may need explicit handling.
How do I prevent duplicate processing in a Spring Boot Kafka consumer?
Use stable event or operation identities and enforce appropriate uniqueness or state-transition rules in durable storage. Coordinate the business update and deduplication record within the relevant database transaction. The implementation should also account for external side effects and crash recovery.
Should I use a dead-letter queue for Kafka consumers?
Dead-letter queues can isolate records that repeatedly fail processing and allow other work to continue. However, moving a required event aside may violate ordering assumptions or leave business workflows incomplete. The design needs controlled recovery and visibility into the affected operations.
How do I handle out-of-order Kafka events?
Define the expected state transitions and determine whether events carry authoritative sequence or version information. The consumer can reject, defer, or reconcile events that arrive out of order. The correct behavior depends on whether missing predecessors are required and whether the business permits reordering.
Can optimistic locking solve Kafka ordering problems?
Optimistic locking can detect conflicting concurrent database updates, but it doesn’t automatically determine the correct business order of events. The application still needs state-transition rules or sequence information. Locking and ordering solve related but different problems.
What happens if a Kafka consumer crashes after committing PostgreSQL?
If the database transaction commits but the Kafka offset isn’t durably committed, the record may be delivered again. The application needs to tolerate repeated processing without creating unintended business effects. Durable idempotency or appropriate state-transition validation can help.
How do I handle external API timeouts in Kafka consumers?
Treat timeouts as potentially uncertain outcomes when the external operation may have succeeded. Preserve a stable operation identity and use provider idempotency or reconciliation capabilities where available. Blind retries can create duplicate external effects.
What do Staff Engineer interviewers look for in Kafka architecture questions?
They may evaluate ordering guarantees, idempotency, failure recovery, operational complexity, and how infrastructure behavior affects business correctness. Staff-level discussions can also include migration strategy and cross-team ownership of event contracts. The expectations depend on the organization and role.
What I Would Tell an Engineer Before Their Next Kafka Interview
I would tell them to stop treating Kafka configuration as the center of the design. Partition counts, consumer groups, and offset commits matter, but they don’t define which business transitions are valid. A system can process every Kafka record successfully and still leave orders in impossible states if the application applies events without enforcing its domain rules.
I would also encourage them to examine every retry path. Moving failed events into another topic can improve throughput while changing the order in which related operations are applied. That trade-off needs to be deliberate, especially when events control payments, inventory, shipments, or other irreversible business effects.
For engineers preparing for advanced backend interviews, the Senior Backend Engineer Interview OS provides structured preparation across distributed systems, concurrency, and production reliability. The Senior & Staff System Design Interview Mastery covers broader architectural decisions and difficult technical follow-ups.
For a comprehensive preparation path, the Senior & Staff Engineer Career Accelerator combines 62 advanced engineering systems with 30 days of private asynchronous mentorship. It includes personalized written feedback on technical interviews, architecture decisions, and career development. Engineers who already know their specific preparation gaps may prefer a focused resource, while those seeking broader guidance can evaluate whether the mentorship format fits their circumstances.
Devrim Özçay — ProdRescue by Devrim
I publish advanced backend engineering investigations, Kafka and distributed systems analysis, and Senior/Staff interview preparation on Substack. The articles examine the engineering decisions that determine whether event-driven systems remain correct when retries, concurrency, and partial failures challenge the original design.
