Migrating from ingress-nginx to OpenResty + WAF, Without Downtime

Thushara Sampath

11 min read

·

Oct 5, 2026

--

Listen

Share

Press enter or click to view image in full size

We recently replaced our NGINX Ingress setup with an OpenResty + WAF-based setup.

I’ve compiled the lessons learned, engineering decisions, and tools I built during the migration into a blog post. It covers the ingress porting process, parity testing, performance tuning, and the production cutover mechanism we used.

Hopefully, it will help anyone planning a similar migration.

Note: This is a fairly detailed post. Feel free to jump to the sections that are most relevant if you’re looking for something specific, or read through the whole post for the full migration story.

Introduction

Every request entering our control plane went through ingress-nginx. It worked. It routed traffic, terminated TLS, and ran reliably.

But there were two problems:

• It didn’t inspect the traffic. No Web Application Firewall (WAF) sat at the control-plane edge.
• At the same time, ingress-nginx was approaching its end of life in March 2026.

So we had two problems that could be solved with one migration: Replace ingress-nginx with a supported edge proxy with a built-in WAF.

This is how we approached it.

The plan

We split the migration into four steps:

1. Build the new edge proxy with WAF
2. Port the ingresses: Match the new edge proxy’s routing behavior with the existing NGINX edge.
3. Benchmark the edges: Compare the performance of the existing and new edge proxies.
4. Cut over the traffic: Move production traffic from NGINX to the new edge without an outage.

This separation turned out to be one of the most important decisions we made. We didn’t want to discover an ingress compatibility problem while we were moving production traffic.

So the strategy was simple: First, make the new edge predictable. Then move the traffic.

Step 1: Building the new edge

The replacement edge was built on OpenResty.

For the WAF, detection and blocking were not the only requirements. We wanted the WAF to run directly within the edge proxy, without introducing another network hop or failure domain. We also needed rule IDs that could be traced when legitimate application traffic triggered a rule.

We evaluated several WAF approaches, and ModSecurity v3 fit these requirements well. We use the OWASP Core Rule Set (CRS) for the WAF rules.

Step 2: Porting ingresses

Don’t migrate everything at once

There were many ingresses, each with its own routing, annotations, TLS configuration, and upstream behavior.

Moving them all at once would have created a debugging nightmare. So in the development environment (DEV), we took a different approach.

We introduced a traffic-switching layer that allowed us to move one ingress at a time.

Press enter or click to view image in full size

Press enter or click to view image in full size

Migrated ingresses went through OpenResty. Everything else continued through nginx.

This gave us a controlled migration loop:

Press enter or click to view image in full size

Press enter or click to view image in full size

Instead of asking:

“Does OpenResty work with our platform?”

we could ask:

“Does OpenResty correctly handle this ingress?”

That made every problem smaller. And it meant other developers could continue using the rest of the DEV environment while we migrated and tested one ingress at a time.

An Ingress is not the whole configuration

One of the first observations was that simply translating an Ingress object into an OpenResty configuration wasn’t enough.

• Some behavior came from annotations.
• Some came from Kubernetes Services.
• Some came from TLS configuration.
• Some came from network policies.

And some only became visible when a real browser or backend exercised the route.

A few examples:

Host header differences

One backend depended on an upstream-vhost annotation.

The route itself looked correct, but the backend received the wrong Host header.

The fix was to explicitly preserve the expected upstream host.

Upstream protocol differences

One backend rejected the new edge with:

426 Upgrade Required

The reason was an HTTP/1.0 → HTTP/1.1 difference between the proxies.

Failure behavior matters too

One route returned a 502 from both edges. At first glance, that looked like parity. It wasn’t.

nginx failed in about a second. OpenResty kept the connection open for around 15 seconds. Same status. Very different behavior under load.

That led to a useful rule:

When comparing two proxies, compare failure behavior, not just successful responses.

Curl isn’t enough

Our curl-based checks passed. A browser exposed CORS preflight differences.

Across the migrated hosts, configuration drift was common enough that browser validation became part of the migration process.

Make parity a first-class test

Once we saw how many small differences could exist between the two edges, manually checking each ingress no longer made sense.

So we built a parity checker.

For every route, it sends the same request to both edges and compares their behavior.

Every failed comparison produces evidence files, so the difference can be investigated without repeatedly reproducing the request.

Press enter or click to view image in full size

Press enter or click to view image in full size

That changed migration validation from:

"Looks okay"

to:

Route → test → evidence → verdict

This was much more useful operationally. Instead of relying on someone remembering what they saw during a manual test, every difference became an artifact that could be investigated and discussed.

What we achieved

By moving ingress by ingress, we were able to discover compatibility problems while the old edge was still serving the rest of the traffic.

More importantly, we learned what information was actually required to port an ingress safely.

That led to the next improvement.

Turn migration knowledge into a tool

Once we understood the migration process, we didn’t want to manually repeat the same investigation for every environment.

So we built a read-only collector.

One run produces the information needed to build OpenResty vhosts:

• hosts and paths
• routing annotations
• backend Service and pod ports
• network-policy information
• TLS configuration
• certificate coverage
• how secrets are produced
• relevant namespace and load-balancer information

The important idea wasn’t the script itself.

It was this:

If a human needs to look in five different Kubernetes objects before creating a vhost, that lookup should probably become a tool.

The collector turned a collection of manual investigations into a repeatable, single-run change request and reduced the need for repeated SRE checks across different Kubernetes objects.

Step 3: Benchmarking the new edge

Before moving further with the migration, we wanted to understand the performance characteristics of the new edge.

We didn’t want to compare the two proxies under different environments and draw conclusions from noisy results.

So we set up a controlled test environment and ran nginx and OpenResty side by side under the same conditions.

We measured:

• throughput
• latency
• concurrent users
• run-to-run variance

The first comparison showed significant latency on the WAF-enabled OpenResty setup.

That gave us a question to answer:

How much of the overhead comes from the WAF, and how much comes from the proxy itself?

So instead of treating OpenResty + WAF as one black box, we broke the test into stages.

Press enter or click to view image in full size

Press enter or click to view image in full size

This separated the cost of the proxy from the cost of WAF processing.

What the WAF actually costs

The results confirmed one expected behavior:

ModSecurity with the OWASP Core Rule Set is slower than a plain proxy. With the WAF enabled, throughput was roughly 7–8× lower in our saturation tests.

That is expected. Every request now goes through the ModSecurity engine and is evaluated against the Core Rule Set across the URI, headers, and request body.

That is CPU work that didn’t exist in the nginx-only configuration. So the WAF overhead itself wasn’t the surprising part.

But then we noticed something more interesting. Even with the WAF disabled, OpenResty was slower than nginx. We expected the opposite.

That meant there was something else to investigate.

Configuration matters

We compared OpenResty’s nginx.conf with the existing ingress-nginx configurations.

We exec’d into the nginx pod and dumped the nginx configuration generated by the ingress controller. We compared it with the OpenResty configuration.

That exposed several differences.

Two were particularly important:

worker_processes auto;

and:

keepalive_requests 100;

Both looked reasonable at first. They weren’t ideal for our containerized workload.

The keepalive_requests change was straightforward.

We increased it from 100 to 1000. This improved throughput by roughly 12–29%, depending on concurrency.

The less obvious problem: worker processes

worker_processes required more investigation.

The OpenResty configuration by default used:

worker_processes auto;

In nginx, auto sets the worker count from the number of CPUs the host reports as online.

It does not look at the container’s CPU limit. Nothing in that calculation knows a quota exists.

That is generally useful on a dedicated machine. It can be a poor fit for a container with a small CPU quota.

For example, a pod running on a 16-core node might only have a fraction of one CPU allocated to it.

With:

worker_processes auto;

the proxy still creates workers for all of the node’s cores.

Multiple workers then compete for a very small CPU allocation, while the Linux Completely Fair Scheduler (CFS) throttles the container when it reaches its quota.

In our case, OpenResty was using less than one CPU.

So we changed it to:

worker_processes 1;

The result was significant.

At 16–32 concurrent clients, throughput improved by roughly 64–78% in the lab. We later saw an 8.8% improvement in production at 30 concurrent clients.

Press enter or click to view image in full size

Press enter or click to view image in full size

The important point was that the WAF wasn’t the only performance factor.

Step 4: The production cutover

By this point, all ingresses had been ported and validated. Now we had a different problem.

How do we move production traffic?

The DEV switching mechanism was excellent for progressive migration. But for production, we wanted something simpler.

We didn’t want production traffic continuously deciding between two different edge implementations.

Instead, we moved the cutover point outside Kubernetes.

The cutover architecture

The old Nginx edge and the new OpenResty edge run side by side in the same cluster.

Each had its own LoadBalancer Service.

A NAT rule in front of the cluster decides which of those load balancers the public address points at.

The architecture therefore looked like this:

Press enter or click to view image in full size

Press enter or click to view image in full size

The nginx edge stays deployed the whole time. It just has no traffic.

The entire production migration became:

Before:
Public address → nginx LB

After:
Public address → OpenResty LB

That’s it. No DNS migration. No application change. No customer change. No deployment.

Why this makes rollback easy

The old edge stays alive. The new edge has its own load balancer. The public address doesn’t change.
So rollback is simply:

Public address
    ↓
Old nginx LB

This gives us a useful property:

The rollback path is simpler than the migration path.

That’s exactly what you want for a high-risk infrastructure change.

Make the cutover measurable

A near-zero-downtime design is only useful if you can actually prove what happened.

So we built a cutover detector.

It continuously probes the edge and identifies which implementation answered the request.

For example, during one migration, one port could already be running through OpenResty while another was still using nginx.

A single “migration complete” metric would hide that. So we measured each path independently, across the public and internal edges and the HTTP/HTTPS ports involved in the migration.

The detector runs for:

• public/internal edges (we replaced two edges in this effort)
• HTTP/HTTPS ports

Press enter or click to view image in full size

Press enter or click to view image in full size

Detector output while switching system ingress https port

WAF blocking comes later

The migration itself used DetectionOnly. That was intentional.

Deploying the WAF and enabling blocking are two different risk decisions.

With DetectionOnly we could get the new edge into production and watch what the Core Rule Set actually matched.

To make those matches visible, we built a Grafana dashboard showing:

• rule hits over time
• rule families being triggered
• the individual rules firing most frequently
• unusual spikes in activity

Press enter or click to view image in full size

Press enter or click to view image in full size

Then we tuned the rules. Some rules needed to be adjusted or excluded for legitimate application traffic, reviewing what matched, understanding why it matched, and validating each change against real requests. That is a topic worth its own blog, so we will leave the details there.

Only after the rules were tuned and we understood the traffic did we enable blocking.

Deploy OpenResty + WAF
          ↓
DetectionOnly
          ↓
Monitor production traffic
          ↓
Build WAF dashboard
          ↓
Tune CRS rules
          ↓
Validate
          ↓
Enable blocking

This separation reduced the risk of turning a security control into an availability problem. “The WAF is deployed” and “the WAF is actively protecting production traffic” became two deliberate milestones rather than one risky switch.

What we achieved

The final migration gave us three things.

1. A new security layer

The control-plane edge now runs OpenResty with ModSecurity v3 and the OWASP Core Rule Set.

The WAF was first deployed in DetectionOnly mode, monitored and tuned against real traffic, and then moved to blocking mode.

2. A repeatable migration process

The migration evolved from manual vhost conversion into a sequence anyone can run for the next environment.

Press enter or click to view image in full size

Press enter or click to view image in full size

The collector, parity checks, performance methodology, and WAF monitoring can be reused for future environments.

3. A low-risk production cutover method

The old and new edges ran side by side. Production traffic moved by changing the NAT destination. Rollback was the same operation in reverse. The production migration completed without a customer-visible outage.

The real lesson: make the change smaller

The interesting part of this migration wasn’t replacing nginx with OpenResty. It was the way we reduced the size of every decision.

Instead of:

“Let’s replace the production ingress.”

we did:

“Let’s migrate one ingress.”

Instead of:

“Let’s enable the WAF.”

we did:

“Let’s observe what the WAF sees.”

Instead of:

“Let’s switch production.”

we did:

“Let’s verify both edges, switch one path, measure it, then continue.”

And instead of:

“Let’s make the rollback work.”

we designed the system, so rollback was simply “change the destination back.”

The migration became a sequence of small, measurable changes rather than one large leap.

If you’re doing the same migration

A few things are worth taking away.

1. Migrate progressively

If you have many ingresses, don’t make the first test a platform-wide cutover.

Move one route at a time.

2. Build a collector early

Find out what actually controls each ingress before you start writing the replacement configuration.

3. Automate parity

Compare status, headers, and response bodies. And compare failures too.

4. Test with a browser

Some differences simply don’t appear in curl.

5. Keep old and new edges alive

A rollback that requires rebuilding the old edge is not a great rollback.

6. Put the cutover outside the system you’re changing

An external VIP/NAT switch gives you a simple, independent traffic-control point.

7. Measure the cutover

Don’t rely on “we didn’t notice an outage.” Have a detector that tells you exactly what happened.

8. Benchmark before tuning

First establish a clean baseline. Then change one variable at a time.

9. Deploy the WAF before enforcing it

DetectionOnly gives you a chance to understand real traffic, build visibility, tune the rules, and validate the configuration before the WAF gets permission to reject requests.