Scaling Event Consumption: Why We Moved from Event Hub to Kafka

Prachi Patil

4 min read

·

Sep 4, 2026

--

Listen

Share

As our event-processing workload grew to millions of events, our existing Event Hub-based architecture started showing scalability limitations.

We were using a Standard Event Hub with 32 partitions. The setup worked well initially, but as the event volume increased, consumer lag started building up and processing latency gradually increased from seconds to, in some cases, minutes.

At first, our natural approach was to scale the number of consumer pods. However, we soon realized that the number of pods was not the only factor determining how much data we could consume.

The real constraint was the 32 partitions behind those consumers.

The Scaling Problem

Our initial architecture looked like this:

Event Producers
                    │
                    ▼
          ┌──────────────────┐
          │    Event Hub     │
          │   32 Partitions  │
          └────────┬─────────┘
                   │
        ┌──────────┼──────────┐
        ▼          ▼          ▼
      Pod 1      Pod 2      Pod 32
        │          │          │
        └──────────┼──────────┘
                   ▼
            Event Processing

As the event volume increased, consumers had more data to process and the backlog started growing.

Add more consumer pods.

But with 32 partitions, there was a practical limit to how much parallelism we could achieve.

Once we reached around 32 consumer pods, adding more pods did not provide meaningful additional throughput because there were no additional partitions to distribute the workload across.

For example, scaling from:

32 pods → 50 pods → 100 pods

does not automatically provide the same increase in consumption capacity.

The Event Hub still had only 32 partitions.

Beyond 32 pods, increasing the number of consumer pods did not make sense for our workload because the partition count was the limiting factor.

The Bottleneck Wasn’t Just the Pods

As the event volume continued to grow, we started seeing pressure at multiple levels:

• 32 partitions limited consumer parallelism
• Increasing ingress and egress traffic added further pressure
• The processing rate could not keep up with the incoming event rate
• Consumer lag continued to increase

This resulted in consumption latency increasing from seconds to seconds/minutes under heavy workloads.

At this point, simply scaling the application layer was no longer enough. We needed more parallelism at the event-stream level.

Moving to Kafka

We decided to move the workload to Kafka with 200 partitions.

The primary reason was straightforward:

We needed significantly more partitions to provide more consumer parallelism.

Event Producers
                    │
                    ▼
          ┌──────────────────┐
          │      Kafka       │
          │  200 Partitions  │
          └────────┬─────────┘
                   │
        ┌──────────┼──────────┐
        ▼          ▼          ▼
     Consumer   Consumer   Consumer
       Pod        Pod         Pod
        │          │          │
        └──────────┼──────────┘
                   ▼
            Event Processing

Moving from 32 to 200 partitions gave us significantly more room to distribute the workload across consumers.

Consumer Concurrency Added Another Level of Scaling

Kafka also gave us the ability to configure consumer concurrency in our application.

This is important because scaling is not only about the number of pods. We can also control how many consumers run concurrently within those pods.

For example, with:

• 200 Kafka partitions
• 10 pods
• 10 consumer threads per pod

we could have up to 100 consumers processing partitions concurrently.

Kafka
                200 Partitions
                      │
        ┌─────────────┼─────────────┐
        ▼             ▼             ▼
      Pod 1         Pod 2         Pod 10
   10 consumers   10 consumers   10 consumers
        │             │             │
        └─────────────┼─────────────┘
                      ▼
              Parallel Processing

The important point is that concurrency works together with partition count.

If we had only 32 partitions, increasing the number of consumer threads to 100 would not give us 100-way partition-level parallelism. There would still be only 32 partitions available to consume.

With 200 partitions, however, we have much more room to increase consumer concurrency and utilize additional processing capacity.

This gave us two useful scaling levers:

1. Increase the number of pods

2. Increase consumer concurrency within the pods

Both are ultimately supported by having enough partitions to distribute the workload.

The Result

The move to Kafka gave us significantly more consumer parallelism and improved our ability to handle increasing event volumes

Press enter or click to view image in full size

Key Takeaway

Our experience highlighted an important principle in high-volume event processing:

Consumer scalability is ultimately constrained by the number of partitions available in the event stream.

With our original Event Hub setup, 32 partitions meant that scaling beyond roughly 32 consumer pods did not provide meaningful additional parallelism.

Kafka’s 200 partitions, combined with configurable consumer concurrency, gave us significantly more flexibility to distribute and process events in parallel.

The migration therefore wasn’t simply about changing the messaging platform. It was about giving the application enough partition-level and consumer-level parallelism to keep up as the workload grew to millions of events.