Migrating Ingress NGINX Controller to istio in Kubernetes environment

Introduction

Migrating ingress traffic in a Kubernetes environment requires careful planning, especially when moving from a traditional ingress controller like Ingress NGINX Controller to a full service mesh such as Istio.

Many Kubernetes environments rely on ingress-nginx as the primary entry point for external traffic. It provides a simple and reliable way to expose services using Kubernetes ingress resources. With the retirement of Ingress-NGINX in our environment, we needed to migrate to a new ingress solution. Instead of replacing it with another traditional ingress controller, we chose to adopt Istio and explore its service mesh–based traffic management capabilities. P.S. https://github.com/kubernetes/ingress-nginx#ingress-nginx-retirement

In this article, I’ll walk through migration from Ingress NGINX to Istio, covering the architectural decisions, migration strategy, challenges encountered, and debugging techniques that helped us successfully transition.

Table of contents

1. Existing architecture short overview
2. Istio Deployment Model Decision
3. Resource Planning for Istio
4. Architecture After Introducing Istio
5. Ingress Configuration vs Istio Configuration
6. Debugging Challenges
   — 503 Error
   — rewrite in istio
   — Regex in istio vs ingress NGINX

Existing Architecture short overview

Before introducing Istio, the Kubernetes environment used a traditional ingress-based architecture to expose services to external traffic. This setup relied on an ingress controller to act as the entry point into the cluster and route incoming requests to the appropriate services.

At a high level, external traffic first reached a cloud-managed load balancer that was automatically provisioned by Kubernetes through a LoadBalancer-type Service. This service exposed a public IP address that served as the external endpoint for all incoming requests. The load balancer forwarded this traffic to the ingress controller running inside the cluster.

The ingress controller was responsible for handling HTTP and HTTPS traffic and applying routing rules defined through Kubernetes ingress resources. These ingress resources defined how requests should be routed based on hostnames and paths. Each microservice that needed to be exposed externally had its own ingress configuration specifying the routing rules for that service.

Internet
   |
Cloud Load Balancer (Public IP)
   |
Kubernetes Service (Type: LoadBalancer)
   |
Ingress Controller
   |
Ingress Resources
   |
Kubernetes Services
   |
Pods (Microservices)

Istio Deployment Model Decision

Istio provides two deployment models:

• Sidecar mode, which deploys an Envoy proxy along with each pod that you start in your cluster, or running alongside services running on VMs.
• Ambient mode, which uses a per-node Layer 4 proxy, and optionally a per-namespace Envoy proxy for Layer 7 features.

Note: If the architecture requires advanced Envoy customization — such as inserting custom HTTP filters, modifying listeners, changing routing behavior, or integrating custom security filters — sidecar mode is required. This is because these capabilities rely on the EnvoyFilter API, which is not supported in ambient mode.

Resource Planning for Istio

• Istio sidecar mode vs Istio Ambient mode resource usage

Before introducing a service mesh like Istio into a Kubernetes environment, it is important to account for the additional resources required by the mesh components. Istio introduces a control plane (primarily istiod) responsible for configuration management, service discovery, and certificate handling, along with an Istio Ingress Gateway that manages incoming traffic to the cluster. When using sidecar mode, each application pod also includes an Envoy proxy container that intercepts and manages traffic, which adds CPU and memory overhead per pod. In contrast, ambient mode reduces this overhead by removing sidecars and handling traffic interception through node-level components such as ztunnel, resulting in a lower overall resource footprint but with a newer architecture and fewer extensibility options compared to sidecar mode.

• Istio vs Ingress Nginx

When comparing Istio with the Ingress NGINX Controller in a Kubernetes environment, the primary difference in resource usage comes from how traffic management is implemented.

An NGINX Ingress Controller typically runs as a small set of pods that handle external traffic routing for the entire cluster. Since routing and TLS termination are centralized in the ingress controller, application pods themselves do not run additional proxy containers. As a result, the overall resource footprint is relatively small and predictable, usually limited to the CPU and memory consumed by the ingress controller pods.

In contrast, Istio introduces a service mesh architecture that includes both control plane components and data plane proxies. The control plane (istiod) manages configuration, service discovery, and security policies, while traffic management is handled by Envoy proxies. In sidecar mode, an Envoy proxy runs alongside each application pod, intercepting inbound and outbound traffic. This provides advanced capabilities such as traffic shaping, mTLS, and detailed observability, but it also increases resource consumption due to the additional proxy containers running across the cluster. In ambient mode, this overhead is reduced by shifting traffic interception to node-level components instead of per-pod sidecars.

Overall, while NGINX Ingress offers a lightweight solution focused primarily on north–south traffic routing, Istio provides a richer set of service mesh features at the cost of higher resource usage due to its distributed proxy architecture.

Architecture After Introducing Istio

Press enter or click to view image in full size

External traffic now enters the cluster through the Load Balancer Service to Istio Ingress Gateway, which acts as the entry point for all incoming requests. Traffic routing is defined using Istio resources such as Gateway and VirtualService, which determine how requests should be routed to the appropriate services. Additionally, DestinationRules define policies such as load balancing and connection behavior for the target services.

Within the cluster, each application pod runs alongside an Envoy sidecar proxy. These proxies intercept inbound and outbound traffic for the application containers and enable service-to-service communication through the mesh. All traffic between services flows through these Envoy proxies, allowing Istio to enforce routing policies, security controls, and observability across the entire service mesh.

Communication flow in istio

User
  ↓
LoadBalancer (Public IP/ Private IP)
  ↓
Istio Ingress Gateway
  ↓
VirtualService routing
  ↓
DestinationRule policies
  ↓
Envoy sidecar (source)
  ↓
Envoy sidecar (destination)
  ↓
Application Pod

Ingress Configuration vs Istio Configuration

• Ingress NGINX config

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: sample-app
spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /app
        pathType: Prefix
        backend:
          service:
            name: app-service
            port:
              number: 80

• Istio equivalent config

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: sample-gateway
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - example.com

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: sample-app
spec:
  hosts:
  - example.com
  gateways:
  - sample-gateway
  http:
  - match:
    - uri:
        prefix: /app
    route:
    - destination:
        host: app-service
        port:
          number: 80

Debugging Challenges

This section will be updated periodically as more debugging scenarios and solutions are identified.

Priority Order for istio resources

The priority order of the istio resorces is as follows
It will look for the resource in the current namespace and that will be applied only and only to that namespace if not present default values
Next it will check istio namespace that will be applied globally to all the other namespaces also and if it is present in both max value will be considered.

Override parameters using envoy filter max_request_bytes, per_connection_buffer_limit_bytes

If the connection is failing due to buffer reached
http inspector: reached max buffer without determining HTTP version, dropping connection thread=20

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: dummy-envoy-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      <labels>
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_INBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.buffer
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.buffer.v3.Buffer
          max_request_bytes: 1073741824 #1024 MB
  - applyTo: HTTP_FILTER
    match:
      context: SIDECAR_OUTBOUND
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.filters.http.buffer
        typed_config:
          '@type': type.googleapis.com/envoy.extensions.filters.http.buffer.v3.Buffer
          max_request_bytes: 1073741824 #1024 MB
  - applyTo: LISTENER
    match:
      context: SIDECAR_OUTBOUND
    patch:
      operation: MERGE
      value:
        per_connection_buffer_limit_bytes: 1073741824 #1024 MB
  - applyTo: LISTENER
    match:
      context: SIDECAR_INBOUND
    patch:
      operation: MERGE
      value:
        per_connection_buffer_limit_bytes: 1073741824 #1024 MB

Istio also uses this acr for charts not only the one mentioned in the docs

oci://gcr.io/istio-release/charts
#To verify
helm show chart oci://gcr.io/istio-release/charts/base --version <istio-version>

If any of the client used http1.0 istio will reject the <http1.1 requests

To check if it is failing due to http version

curl --http1.0 -v POST <request parameters>

Adding this will make http1.0 work

apiVersion: networking.istio.io/v1beta1
kind: ProxyConfig
metadata:
  name: allow-all-http
  namespace: istio-system
spec:
  environmentVariables:
    ISTIO_META_HTTP10: "1"

Destination rules with the parameters that mey get missed

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata:
  name: dummy-dr
  namespace: istio-system
spec:
  host: <host-name>
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL/DISABLE
    connectionPool:
      http:
        http1MaxPendingRequests: 1024
        idleTimeout: 300s
        maxRequestsPerConnection: 1 (0 for no limit)
      tcp:
        connectTimeout: 300s
        maxConnections: 1024

More Debugging Techniques

• To enable debug logs for a pod’s istio proxy

istioctl proxy-config log  <pod-name> -n <namespace> --level debug

• To check destination rule parameters when one istio-proxy pod talks to other pods

istioctl proxy-config cluster <source-pod-name> -n <source-namespace> --fqdn <destination-full-service-host> -o json | jq '.[].circuitBreakers, .[].connectTimeout, .[].maxRequestsPerConnection'

• To check the envoy filter parameters for a istio-proxy

istioctl proxy-config listeners <pod-name> -n <namespace> -o json | grep <parameter you want to search>

503 the most frustrating error to debug

• Check all the resources are accessible to istiod and istio-gateway preferably in the same namespace.
• There are no networkpolicies blocking the communication between istiod, gateway and the proxies

kubectl get networkpolicy -A

kubectl describe networkpolicy <policy-name> -n <istio-namespace>

• Check the VirtualService and Destination rule configuration using istioctl
  — To install istioctl
  — Now use istioctl analyze command to check if the configurations are valid or not

istioctl analyze --all-namespaces

• Check whether the gateway proxy received the routing configuration using istioctl proxy command

istioctl proxy-config routes <ingress-gateway-pod> -n <istio-namespace>

istioctl proxy-config clusters <ingress-gateway-pod> -n <istio-namespace>

• Start debugging from the application pod itself to confirm the pod is running correctly.
  Exec into the pod and test the application locally using localhost.
  If the application does not respond on localhost, the issue is likely with the application itself rather than Istio or Kubernetes networking.

# To get the pod
kubectl get pods -n <namespace>

# Exec to the Pod
kubectl exec -it <pod-name> -n <namespace> -- /bin/sh

# Once inside the pod
curl http://localhost:<port>
or
wget -qO- http://localhost:<port>

• Check the port for the service and pod match
  If the targetPort does not match the port exposed by the pod, the service will not route traffic correctly.

# To check the svc port
kubectl get svc <service-name> -n <namespace>
kubectl describe svc <service-name> -n <namespace>

# To check the pod port
kubectl get pod <pod-name> -n <namespace> -o yaml | grep -A2 containerPort

• Confirm the service is reachable inside the cluster, create a temporary debugging pod and attempt to access the service endpoint.

To create and exec into the pod
kubectl run debug-pod --rm -it --image=busybox -- /bin/sh

Run this command to check the svc is reachable
curl http://<service-name>.<namespace>.svc.cluster.local:<port>/<endpoint-to-check>

rewrite in istio

When configuring URL rewrites in Istio on Kubernetes, a common issue occurs when trying to rewrite a prefixed path to the root of the backend service.

The request uri’s:

/foo/bar
/foo/bar/subpath
/foo/bar/subpath/test

Rewritten to:

/
/subpath
/subpath/test

A more reliable approach is to define two match rules — one for the exact path and another for the prefixed path.

http:
- match:
  - uri:
      exact: /foo/bar
  - uri:
      prefix: /foo/bar/
  rewrite:
    uri: /
  route:
  - destination:
      host: some-service
      port:
        number: 80

Regex in istio vs ingress NGINX

NGINX uses the PCRE (Perl Compatible Regular Expressions) engine, while Istio uses RE2, which is designed to be safer and faster but supports a smaller feature set.
A regex like this may work in NGINX but fail in Istio:

^(?!/admin).*\.png$

This uses negative lookahead, which RE2 does not support.

Example: Ingress-NGINX Regex Rule Migration

• In ingress-nginx, the following rule was used to route image and static file requests while excluding certain paths.
  This rule uses negative lookahead to exclude specific paths such as /exchange and /auth.

spec:
  rules:
  - host: example.com
    http:
      paths:
      - path: /(?!(exchange|auth|image))/.*\.(jpg|jpeg|png|gif)$
        pathType: ImplementationSpecific
        backend:
          service:
            name: image-service
            port:
              number: 80

Problem in Istio: The same regex cannot be used directly in Istio because RE2 does not support lookahead expressions. Attempting to use this pattern in a VirtualService results in routing failures or configuration validation errors.

• Instead of using negative lookaheads, the routing logic can be implemented using multiple match rules.

http:
- match:
  - uri:
      regex: "/.*\\.(jpg|jpeg|png|gif)$"
  route:
  - destination:
      host: image-service
      port:
        number: 80

• If certain paths must be excluded, they can be handled using higher-priority match rules.
  Istio evaluates rules in order, so specific routes should appear before broader regex matches.

http:
- match:
  - uri:
      prefix: "/exchange"
  route:
  - destination:
      host: exchange-service

- match:
  - uri:
      prefix: "/auth"
  route:
  - destination:
      host: auth-service

- match:
  - uri:
      regex: "/.*\\.(jpg|jpeg|png|gif)$"
  route:
  - destination:
      host: image-service

Press enter or click to view image in full size