Member-only story
We Replaced Kafka With 1 PostgreSQL Table: No Measurable Consumer Lag
Three Brokers, One Tired Engineer, And A Workload That Never Needed Either Of Them
Kafka woke me up four times in one month, and our peak traffic was 400 messages per second.
If you have ever stared at a broker dashboard in the dark, wondering why a team of nine needs a distributed log, you already know this feeling.
We had three brokers, a schema registry, a lag dashboard, and exactly one person who understood all of it. That person was on vacation.
How We Got Here
Growth did come, but in customers, not in events. We peaked at 400 events per second and idled at 60 most afternoons.
The real pain lived somewhere else. Every order wrote to Postgres, then published to Kafka, and twice a month those two steps disagreed.
We would find an order with no event, or worse, an event with no order. Explaining that to a customer never got easier.
+---------+ +------------+
| Order |----->| Postgres |
| Service | +------------+
| | +------------+ +-----------+
| |----->| Kafka |----->| Workers |
+---------+ +------------+ +-----------+…
