The Outbox Pattern, Done Properly: Beyond the Naive Implementation
The Code That Looks Right
Here’s a Spring Boot method you’ve probably written, reviewed, or shipped:
@Transactional
public Order placeOrder(OrderRequest request) {
Order order = orderRepository.save(new Order(request));
kafkaTemplate.send("orders", new OrderPlacedEvent(order));
emailService.sendConfirmation(order);
return order;
}It looks defensive. There’s a @Transactional. The order is saved before the event is published. If anything throws, surely the transaction rolls back and we're safe.
It’s broken in two ways. Neither will show up until the day you can’t afford for them to. This article is about both — and the pattern that fixes them.
Problem 1: The Dual-Write Race
The first failure mode is the famous one, but most explanations of it are vague. Let’s be precise.
When placeOrder exits, Spring's transaction manager issues COMMIT to the database. That commit is a network round trip.
The Kafka send is also a network call — but it doesn’t work the way you might think. kafkaTemplate.send returns immediately. It doesn't wait for the broker. The message sits in a buffer inside the producer and gets sent later, on a background thread.
And there is no shared transaction between the database and Kafka. They’re separate systems. The database can’t roll back a Kafka send, and Kafka can’t roll back a database commit.
The actual timeline looks like this:
1. INSERT INTO orders (...)
2. kafkaTemplate.send("orders", event) ← buffered in the producer
3. COMMIT
4. ...method returns
5. Kafka producer flushes the buffer ← can fail hereIf step 5 fails — broker rebalance, network glitch, leader election, broker crash — the row is in the database, the customer got a 200, and the event is gone. The inventory service will never know that order existed.
Flip the order, send to Kafka first, and you get the inverse failure: event published, transaction rolled back, ghost orders downstream.
You can’t fix this by reordering the calls. You can’t fix it with retries inside the method — the method has already returned. You can’t fix it with @Transactional(rollbackFor = ...), because Kafka was never part of the transaction.
This is the dual-write problem. The only real fix is to stop doing two writes.
Problem 2: The @Transactional Trap
The second failure is the one nobody warns you about. It’s worse than the first, because it doesn’t just lose events — it takes the whole service down.
Look at the original method again. @Transactional wraps the entire body. That means a database connection is borrowed from the pool when the method starts and returned only when it finishes. Now ask: how long does that method take?
orderRepository.save— milliseconds.kafkaTemplate.send— usually milliseconds, but it can block for up to two minutes if the broker is unhealthy.emailService.sendConfirmation— hits SMTP or an external API. Tens of milliseconds on a good day, several seconds on a bad one. Or it just hangs until the HTTP client times out.
While the email service hangs, your database connection is held. Held by a method that doesn’t need the database anymore. It’s just waiting on Gmail.
HikariCP defaults to 10 connections. You serve 10 concurrent orders during a spike. SendGrid has a slow afternoon. Now all 10 connections are stuck on placeOrder calls waiting for SMTP. The 11th request to anything in your service — a health check, a GET /orders/{id}, an unrelated endpoint — waits on the pool. Then times out. Then the load balancer marks the pod unhealthy. Kubernetes restarts it. The new pod warms up, accepts traffic, and immediately runs into the same wall.
This is a self-inflicted outage caused by a single annotation in the wrong place.
The rule is simple: a @Transactional method does database work. Nothing else. Not Kafka. Not email. Not S3. Not REST calls. If a remote system can hang, it does not belong inside a transaction.
Get Lampis Papantoniou’s stories in your inbox
Join Medium for free to get updates from this writer.
The Outbox Pattern fixes Problem 1. The discipline of keeping @Transactional short fixes Problem 2. You need both.
The Outbox Pattern in One Picture
The naive version writes to two systems and hopes both succeed. The Outbox writes to one system — the database — and lets a separate process do the second write later, with retries.
[ Client ]
│
▼
[ Order Service ] ──── single TX ────▶ [ Postgres ]
├── orders
└── outbox ← event lives here
│
│ polls
▼
[ Outbox Relay ]
│
▼
[ Kafka ]
│
▼
[ Inventory ], [ Email ], ...Two writes still happen. The trick is that the first one — orders and outbox — happens atomically, in one transaction, to one database. The second one — outbox to Kafka — happens later, with retries, and the source of truth is now the outbox row. If Kafka is down, the row sits there until Kafka comes back. Nothing is lost.
The Schema
CREATE TABLE outbox (
id BIGSERIAL PRIMARY KEY,
aggregate_type VARCHAR(100) NOT NULL,
aggregate_id VARCHAR(255) NOT NULL,
event_type VARCHAR(100) NOT NULL,
payload JSONB NOT NULL,
created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),
published_at TIMESTAMPTZ,
retry_count INT NOT NULL DEFAULT 0,
last_error TEXT
);
CREATE INDEX idx_outbox_unpublished
ON outbox (created_at)
WHERE published_at IS NULL;A few decisions worth defending:
published_at instead of a status column. A nullable timestamp answers two questions with one column: is it published? and when? It's also cheaper to index than a string. (You could store payload as text instead of jsonb if you wanted — the relay doesn't care either way. I prefer jsonb because during incidents you can query inside the payload directly, which has saved me debugging time more than once.)
Partial index on unpublished rows. This is the single most important index in your system. Once published_at is set, the row drops out of the index, so the relay's polling query stays fast even when the table has millions of historical rows.
aggregate_id as the future Kafka partition key. Events for the same order go to the same partition, which keeps them in order. Don't skip this column.
retry_count and last_error. When something stops publishing, you'll want to know why without tailing logs.
Don’t delete rows in the publish path. Keep them for a few days for replay, audit, and debugging, then reap them with a separate cleanup job.
The Producer: Atomic Writes Only
This is where the discipline of Problem 2 pays off. The transactional method writes to two tables in one database. That’s it.
@Service
public class OrderService {
private final OrderRepository orderRepository;
private final OutboxRepository outboxRepository;
private final ObjectMapper objectMapper;
// constructor omitted
@Transactional
public Order placeOrder(OrderRequest request) {
Order order = orderRepository.save(Order.from(request));
OrderPlacedEvent event = new OrderPlacedEvent(
order.getId(), order.getUserId(), order.getTotalAmount()
);
outboxRepository.save(OutboxEvent.builder()
.aggregateType("Order")
.aggregateId(String.valueOf(order.getId()))
.eventType("OrderPlaced")
.payload(toJson(event))
.build());
return order;
}
private String toJson(Object o) {
try {
return objectMapper.writeValueAsString(o);
} catch (JsonProcessingException e) {
throw new IllegalStateException("Failed to serialize event", e);
}
}
}Notice what isn’t here: no kafkaTemplate, no emailService, no HTTP client. Both rows commit together or not at all. The method holds a connection for milliseconds. If serialization fails, the transaction rolls back and the order is never created — which is correct. An order you can't tell anyone about is worse than no order.
The Relay, Done Right
The relay is where most tutorials quietly break. A common version wraps the whole publish loop in @Transactional and waits up to ten seconds for each Kafka send, which means a database connection is held for ten seconds per event, batched fifty at a time. Run that on two pods and you'll also discover both pods publishing the same events, because there's no locking. We're going to fix both.
Three rules:
- The Kafka send happens outside any database transaction.
- Use
SELECT ... FOR UPDATE SKIP LOCKEDso multiple relay instances claim different batches. - The status update is its own short transaction, after the broker acknowledges.
public interface OutboxRepository extends JpaRepository<OutboxEvent, Long> {
@Query(value = """
SELECT * FROM outbox
WHERE published_at IS NULL
ORDER BY created_at
LIMIT :limit
FOR UPDATE SKIP LOCKED
""", nativeQuery = true)
List<OutboxEvent> lockBatchForPublishing(@Param("limit") int limit);
@Modifying
@Query("UPDATE OutboxEvent o SET o.publishedAt = :now WHERE o.id = :id")
void markPublished(@Param("id") Long id, @Param("now") OffsetDateTime now);
@Modifying
@Query("UPDATE OutboxEvent o SET o.retryCount = o.retryCount + 1, o.lastError = :err WHERE o.id = :id")
void incrementRetry(@Param("id") Long id, @Param("err") String err);
}@Component
public class OutboxRelay {
private static final Logger log = LoggerFactory.getLogger(OutboxRelay.class);
private static final int BATCH_SIZE = 100;
private final OutboxRepository outboxRepository;
private final KafkaTemplate<String, String> kafkaTemplate;
private final TransactionTemplate txTemplate;
@Scheduled(fixedDelay = 1000)
public void drain() {
List<OutboxEvent> batch = txTemplate.execute(status ->
outboxRepository.lockBatchForPublishing(BATCH_SIZE)
);
if (batch == null || batch.isEmpty()) return;
for (OutboxEvent event : batch) {
try {
kafkaTemplate
.send("order-events", event.getAggregateId(), event.getPayload())
.get(5, TimeUnit.SECONDS); // outside any DB transaction
txTemplate.executeWithoutResult(s ->
outboxRepository.markPublished(event.getId(), OffsetDateTime.now())
);
} catch (Exception e) {
log.warn("Publish failed for event {} (will retry)", event.getId(), e);
txTemplate.executeWithoutResult(s ->
outboxRepository.incrementRetry(event.getId(), e.getMessage())
);
}
}
}
}Read it carefully. There are three short transactions per batch iteration, not one big one: the lock-and-claim, the per-event status update on success, the per-event retry bump on failure. The Kafka send sits between them, holding no connection.
When two pods run this code, each calls lockBatchForPublishing. Postgres hands the first pod rows 1–100 and locks them, and the SKIP LOCKED clause means the second pod skips those and gets rows 101–200 instead of waiting. They publish in parallel without stepping on each other.
One subtle thing: using aggregateId as the Kafka key keeps per-aggregate order, but parallel relay pods can publish events from different aggregates out of insertion order. That's fine for almost everyone. If you genuinely need global ordering, you need exactly one relay instance — and then you have a different scaling problem.
The Mistakes That Will Still Bite You
Even with everything above correct, three things will get you in production.
Consumers are not idempotent. The Outbox guarantees at-least-once delivery. Kafka can replay. Your relay can crash between sending and marking the row published, and on restart it will resend. Every consumer must dedupe — typically with a processed_events table keyed by event ID, checked at the start of each handler. If your downstream effect is "charge the customer," at-least-once without idempotency means double charges.
The outbox table grows without bound. Even with the partial index, a table with hundreds of millions of rows will eventually slow down inserts and maintenance. Run a daily job that deletes published rows older than your replay window — seven days is a reasonable default. Partition by month if your volume justifies it.
Nobody watches the queue depth. Expose SELECT count(*) FROM outbox WHERE published_at IS NULL as a Prometheus gauge. When Kafka has an outage, this number climbs and your alerting fires. The data is safe in the database, and you know exactly where to look. Without this metric, your first signal will be angry consumers.
TL;DR
Don’t write to a database and a message broker in the same method and hope. Write to the database only, in a short transaction that does nothing else. Have a separate relay drain the outbox to Kafka with row-level locking. Make consumers idempotent. Watch the queue depth.
The Outbox Pattern is not complicated. It just demands you stop pretending two systems can be written to atomically — and stop letting @Transactional hold a connection while you talk to the internet.
