Dynamic Payment Routing at Scale
For a long time, Snitch relied entirely on a single payment provider — our primary payment provider.
It worked well, handled our payment volume reliably, and required very little operational effort. Like many startups, optimising the checkout experience wasn’t our biggest concern initially. Shipping features and growing the business took priority.
But as order volumes grew, one limitation became increasingly obvious.
A single payment provider is a single point of failure.
Whenever Primary Gateway experienced a partial outage, network degradation, or elevated failure rates, our checkout success rates immediately suffered. Even though these incidents weren’t frequent, they were enough to highlight an uncomfortable reality: our payment infrastructure had no redundancy.
In ecommerce, every failed payment is potentially a lost customer.
The solution wasn’t to replace Primary Gateway — it was to stop depending on only one gateway.
That’s when we integrated a secondary payment provider. At first glance, the problem seemed solved. If the primary provider has issues, use the secondary provider. Unfortunately, reality turned out to be much more complicated.
The Real Problem
Adding another gateway is easy.
Deciding which gateway should process every payment is the difficult part.
A simple 50–50 split sounds reasonable until you start asking practical questions.
- How do we safely launch a brand-new gateway?
- Should UPI traffic move before Cards?
- What if Android is stable while iOS still needs validation?
- During a flash sale, can we maintain an exact traffic split under thousands of concurrent requests?
- What happens if one gateway suddenly starts failing?
- Once it recovers, how do we gradually restore traffic without causing another outage?
We quickly realized we didn’t need two payment providers.
We needed a payment routing engine.
Design Goals
Before writing a single line of routing logic, we defined what the system needed to achieve.
The routing engine had to satisfy several constraints simultaneously.
Safe Launches
A newly integrated gateway should never receive 100% of production traffic immediately.
We wanted to start with a very small percentage, observe metrics, and gradually increase confidence.
Independent Routing
Every payment method behaves differently.
UPI might perform exceptionally well while Cards may still require validation.
Similarly, Android, iOS, and Web don’t necessarily exhibit identical behaviour.
Instead of having one global traffic split, routing should be configurable independently for every combination of:
Platform (Web / Android / iOS)
Payment Method (UPI / Cards / Net Banking / Wallets)
This allows, for example,
Android UPI → 30% Secondary Gateway
Android Cards → 0% Secondary Gateway
iOS UPI → 10% Secondary Gateway
Web Net Banking → 50% Secondary Gatewaywithout one configuration affecting another.
Self Correcting
Traffic should naturally stay close to its configured target.
Nobody should have to monitor dashboards throughout the day and manually adjust percentages whenever routing drifted.
Sale Ready
During major sales, checkout traffic increases by several multiples within minutes.
The routing system should continue maintaining its configured split without becoming unstable.
Automatic Failover
If either payment provider experiences an outage or significant degradation, traffic should automatically move toward the healthier gateway.
Equally importantly, once the gateway recovers, traffic should return gradually instead of instantly.
With these principles in mind, the routing engine evolved through four iterations.
V1 — Safe Percentage Rollouts
The first version solved the simplest problem:
How do we safely launch a newly integrated provider?
Every (Platform, Payment Method) pair was assigned a configurable target percentage.
For example,
Android UPI → 10% Secondary Gateway
Android Cards → 0%
iOS UPI → 5%
Web UPI → 20%A target of
- 0% meant all traffic goes to the primary provider.
- 100% meant everything goes to the secondary provider.
Anything in between represented the desired traffic split. This gave us complete flexibility.
Instead of migrating the entire checkout system overnight, we could slowly increase traffic as confidence improved.
Initially, only UPI was enabled while Cards and Net Banking continued using the primary provider.
The First Request Problem
One subtle issue appeared almost immediately.
Suppose Android UPI is configured to send 20% of traffic to Secondary Gateway.
For the very first payment of the day, there are no statistics yet.
No completed payments.
No traffic ratios.
Nothing to compare against.
So how do you decide where this payment should go?
A straightforward solution would be to generate a random number between 0 and 99. If the number is less than 20, route the payment to Secondary Gateway; otherwise, send it to Primary Gateway. While this works over a very large number of requests, it behaves poorly when traffic is low.
For example, the first ten requests of the day could randomly become:
RP
RP
RP
RP
RP
RP
RP
RP
RP
RPEven though Secondary Gateway is supposed to receive roughly 20% of the traffic, randomness doesn’t guarantee that distribution over a small sample. You might see 0%, 50%, or anything in between simply due to chance.
Instead of relying on randomness, we made the routing decision deterministic. Every payment request already has information that’s unique to it, such as the order ID or payment ID. We pass this identifier through a hash function, producing a number that is effectively random but always the same for that input.
The resulting hash is mapped into one of 100 buckets (0–99).
For example:
Order #1001 → Hash → Bucket 07
Order #1002 → Hash → Bucket 82
Order #1003 → Hash → Bucket 14
Order #1004 → Hash → Bucket 63If the configured Secondary Gateway target is 20%, then buckets 0–19 are assigned to Secondary Gateway, while buckets 20–99 go to Primary Gateway.
0 - - - - - - - - - - - - - - - 99| - - - 20% - - - | - - - - - - - - 80% - - - - - - - - |
Secondary Gateway Primary GatewaySo the routing becomes:
Bucket 07 → Secondary Gateway
Bucket 82 → Primary Gateway
Bucket 14 → Secondary Gateway
Bucket 63 → Primary GatewayThis approach has two important advantages.
First, the same order is always routed to the same gateway if the routing decision needs to be retried, eliminating inconsistencies caused by generating a new random number on every attempt.
Second, because hash values are uniformly distributed, the traffic naturally converges toward the configured percentage as requests increase, without relying on pure chance for every individual request.
Get SNITCH Threads’s stories in your inbox
Join Medium for free to get updates from this writer.
V1 gave us safe and predictable traffic rollouts.
However, once traffic started flowing continuously throughout the day, another challenge emerged.
V2 — Self Correcting Traffic Allocation
Static percentage routing works surprisingly well.
Until it doesn’t.
Imagine Android UPI is configured for 20% Secondary Gateway. Throughout the day, random variation slowly causes the actual split to drift.
And, instead of receiving exactly 20% of traffic, Secondary Gateway might gradually receive 24%, or perhaps only 15%.
Small differences accumulate over time.
Instead of making routing decisions purely from configuration, we began comparing configuration against reality.
For every completed payment, we tracked today’s cumulative traffic.
Actual Share = Secondary Gateway Payments/Total PaymentsFor every incoming checkout —
If Secondary Gateway was below target, send the next request to Secondary Gateway, otherwise send it to Primary Gateway
For example,
Configured target: 20%
Current traffic:
Primary Gateway = 76
Secondary Gateway = 19
Actual share : 19 / 95 = 20%Since Secondary Gateway is already meeting its target, the next payment goes to Primary Gateway.
If Secondary Gateway had processed only ten payments instead,
10 / 86 = 11.6%the next request would immediately be routed toward Secondary Gateway until the difference disappeared.
Instead of blindly following configuration, the router continuously corrected itself.
No operator intervention.
No manual percentage changes.
Just continuous convergence toward the configured target.
For normal traffic, this worked remarkably well.
Then came our first major sale.
V3 — When High Concurrency Breaks Good Algorithms
Flash sales exposed a weakness we hadn’t previously encountered.
The router decides before a payment completes, but routing statistics update after the payment finishes.
That delay seems insignificant. Under normal traffic, it is.
During a sale, it becomes a feedback problem.
Imagine a thousand checkout requests arriving simultaneously.
Every one of them observes nearly identical routing statistics because none of the payments have completed yet.
As a result, all thousand requests make nearly identical routing decisions.
Only several seconds later do completed payments begin updating the metrics. By then, the router has already overshot its intended split.
The problem wasn’t the controller.
The problem was stale feedback.
Our original controller also used the entire day’s traffic as its denominator.
After processing several hundred thousand payments, a burst of ten thousand checkouts barely changes the overall ratio.
Ironically, the period where we needed the fastest reaction was exactly when the controller reacted the slowest.
To solve this, we switched from cumulative daily statistics to a rolling time window.
Instead of asking,
What has today’s traffic looked like?
the controller asks,
What has traffic looked like over the last few minutes?
Recent traffic influences routing decisions much more heavily than events from several hours ago.
This dramatically improved responsiveness during traffic spikes while still smoothing normal statistical noise.
To prevent rapid oscillations, we also limited how quickly the effective traffic split could change within a short period.
The result was a controller that remained stable even under several times our normal checkout volume.
V4 — Traffic Allocation Isn’t Enough
The routing engine now maintained its configured traffic split extremely well, but there was still one important blind spot.
The controller only understood volume. It knew nothing about health.
Imagine configuring Secondary Gateway to receive 30% of UPI traffic.
If Secondary Gateway suddenly starts failing every payment, the proportional controller continues sending approximately 30% of requests there because failed payments still count toward traffic volume.
Operationally, this is unacceptable.
Traffic allocation and gateway health are two completely different problems.
So we introduced a second controller. Alongside traffic distribution, we continuously monitor each gateway’s rolling success rate independently for every platform and payment method.
If one gateway’s success rate falls significantly below the other over a recent window, we treat it as a degradation event.
Instead of following the configured target, the unhealthy gateway’s effective allocation is automatically reduced while the healthier gateway absorbs the remaining traffic.
The important detail is that recovery is intentionally slower than failover.
As soon as a degraded gateway recovers, we don’t immediately restore all its traffic.
Instead, we reintroduce it gradually.
A small canary percentage.
Observe.
Increase.
Observe again.
Only after confidence is restored does the gateway return to its configured allocation.
This avoids oscillations caused by gateways repeatedly failing and recovering.
Final Architecture
By this point, the routing engine had evolved into two independent control loops working together.
The first controller continuously keeps traffic close to its configured allocation. The second controller continuously monitors gateway health and overrides traffic allocation whenever reliability is at risk.
Conceptually, every checkout now flows through a pipeline like this:
Both controllers operate independently for every (Platform, Payment Method) combination, allowing Android UPI, iOS Cards, Web Net Banking, and every other routing group to evolve independently.
A problem affecting one payment method never impacts another.
Lessons Learned
Looking back, none of the individual ideas were particularly revolutionary. Percentage routing, rolling windows, proportional control, and health monitoring are all well-understood concepts. The challenge was combining them into a system that behaves predictably under real-world conditions.
Some of our biggest takeaways were:
- A second payment gateway alone does not improve reliability, a routing strategy does.
- Percentage-based routing is sufficient for launches but not for long-running production systems.
- Feedback delays become a real engineering challenge under high concurrency.
- Recent traffic is often more valuable than historical traffic when making routing decisions.
- Traffic allocation and health monitoring should be treated as separate control problems.
- Recovery requires just as much engineering as failover.
Today, the routing engine allows us to onboard new gateways safely, maintain precise traffic allocation during everyday operations, withstand flash-sale traffic spikes, and automatically recover from gateway failures — all without requiring manual intervention.
What began as a simple traffic split eventually evolved into an adaptive control system that continuously balances rollout safety, operational resilience, and checkout success.
