Azure Kubernetes Application Network, Part 3: AI Gateway, Observability and Production Fit
Part 1 introduced AppNet, while Part 2 demonstrated its use in zero-trust and multi-cluster patterns. This concluding section explores a shared AI gateway, the operational signals derived from the mesh, and the architectural tradeoffs that influence whether AppNet should be integrated into your platform.
Series navigation
- Part 1: What AppNet is and how to get started
- Part 2: Zero-trust and multi-cluster patterns
- Part 3: AI gateway, observability and production fit
Example three: a shared AI gateway that rate-limits by application
The third example has made AppNet interesting to platform teams that previously had no interest in service mesh, and Microsoft characterizes it as a first-party pattern. The scenario involves a platform team deploying a single agentgateway within the cluster that interfaces with Azure OpenAI and several other model providers. Numerous internal applications access this gateway from various namespaces. The finance team requires per-application token budgets to prevent a runaway LangChain agent from exhausting the entire OpenAI budget in a weekend.
Without AppNet, this process becomes cumbersome. The typical solution involves issuing an API key for each application, with the gateway imposing rate limits on those keys. Managing key rotation, storing keys securely in Key Vault, injecting them into the correct namespace, and revoking access during incidents all add complexity. This approach only succeeds if every application consistently uses the keys properly.
AppNet modifies the model by leveraging ztunnel-enforced workload identities in all enrolled namespace pods. This allows agentgateway to verify caller identities using SPIFFE IDs from the mTLS handshake, eliminating the need for API keys for identity verification. Rate limiting is also applied on this basis.
The configuration consists of three parts. The first is an AgentgatewayPolicy linked to the Gateway that manages model API requests by enforcing rate limits and using the source namespace as the key.
apiVersion: agentgateway.io/v1
kind: AgentgatewayPolicy
metadata:
name: openai-token-budget
namespace: ai-platform
spec:
targetRefs:
- group: gateway.networking.k8s.io
kind: Gateway
name: agentgateway
rateLimit:
descriptors:
- entries:
- key: source_namespace
value: "{{ source.identity.namespace }}"
- key: model
value: "{{ request.headers['x-model'] }}"
service: ratelimit.ai-platform.svc.cluster.localThe second part is the rate-limit server itself, configured as a ConfigMap that specifies the tokens-per-minute budget for each application. For example, a team with the namespace fraud might have a larger quota than a team in the marketing-experiments namespace.
apiVersion: v1
kind: ConfigMap
metadata:
name: ratelimit-config
namespace: ai-platform
data:
config.yaml: |
domain: openai_tokens
descriptors:
- key: source_namespace
value: fraud
rate_limit:
unit: minute
requests_per_unit: 100000
- key: source_namespace
value: marketing-experiments
rate_limit:
unit: minute
requests_per_unit: 10000
- key: source_namespace
rate_limit:
unit: minute
requests_per_unit: 1000The third part is the AuthorizationPolicy on the gateway, which blocks requests from any entity not explicitly onboarded by the platform team. This prevents overlooked tenants from quietly consuming tokens. According to Istio security guidance, it is better to use explicit SPIFFE principals rather than namespace-only matching, because principals link rules to specific service accounts, not just to entities within a namespace (see the source-matching reference). Including both fields creates a layered defense that fails closed if either component is altered.
apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
name: agentgateway-allowed-tenants
namespace: ai-platform
spec:
targetRefs:
- kind: Gateway
group: gateway.networking.k8s.io
name: agentgateway
action: ALLOW
rules:
- from:
- source:
principals:
- cluster.local/ns/fraud/sa/fraud-agent
- cluster.local/ns/checkout/sa/checkout-agent
- cluster.local/ns/marketing-experiments/sa/marketing-agent
- cluster.local/ns/reports/sa/reports-agent
namespaces: ["fraud", "checkout", "marketing-experiments", "reports"]As a result, finance benefits from predictable per-application spending limits enforced at the network level, while applications benefit from a seamless “just call the gateway” experience. Additionally, the platform team no longer needs to manage API keys. When a new team joins, onboarding is simplified to a pull request that adds their service account principal and namespace to the AuthorizationPolicy and a budget entry to the ConfigMap.
Observability: metrics and logs that are already there
One of the practical yet often overlooked features of AppNet is its ability to provide valuable signals immediately upon enrolling in a namespace (metrics reference, logs reference). Ztunnel offers per-connection telemetry showing mTLS success and failure, while the waypoint provides standard Istio access logs and Prometheus metrics. When you connect AppNet to an AKS cluster with Azure Managed Prometheus and Azure Managed Grafana enabled, these metrics are conveniently available in the same workspace your cluster uses. This means you don’t need extra Prometheus instances or additional scrape configurations, making your monitoring setup simpler and more efficient.
A common alert signal is the rate of L4 mTLS failures per source and destination, indicating when a workload attempts to connect to an unauthorized namespace. Another useful signal is the waypoint’s 4xx rate broken down by target service, which reveals when an application starts sending unexpected traffic, such as a new endpoint or a changed method that your AuthorizationPolicy now blocks. Both metrics usually increase during deployments, and displaying them alongside application metrics in Grafana helps confirm whether everything is functioning correctly, even if the deploy appears successful.
Logs such as control plane events and waypoint access logs can be sent to Azure Monitor Logs via the standard AKS log collection. CloudOps teams already using Container Insights won't need to make any changes; the logs will appear in the same Log Analytics workspace with only minor label differences.
Where Azure Kubernetes Application Network is a good fit
The honest sweet spot is a platform team that manages multiple AKS clusters, prefers default mTLS on east‑west traffic, and has Layer 7 policy needs that go beyond what network policies can express. If your environment hosts workloads from different tenants within the same cluster or has compliance needs requiring identity‑based authorization between services, then AppNet is perfectly suited for your needs.
Multi-cluster topologies work especially well. If you're operating active-active setups across regions or dividing workloads across clusters to minimize blast radius, AppNet eliminates the need for the federated Istio project, which has historically drained resources from many platform teams. Its shared trust domain and managed control plane eliminate the two common problem areas in federated meshes.
AI platforms offering a shared gateway for multiple tenants are a natural fit, as demonstrated by the third example. Treating workload identity as a first‑class concept and enforcing it at the network layer eliminates a category of API‑key rotation issues that are challenging to resolve cleanly through other means.
Regulated workloads see disproportionate benefits. Automated mTLS, identity-based policies, and managed upgrades simplify compliance documentation into a few Kubernetes resources that can be easily presented to an auditor. When asked, “Show me how you prevent service A from talking to service B”, the answer is just two lines.
Where Azure Kubernetes Application Network is the wrong tool
AppNet functions as a service network. It is not an ingress controller, WAF, or a substitute for Application Gateway for Containers or Azure Front Door. If your challenge is to expose a public API with authentication, WAF rules, and TLS termination, AppNet alone is not sufficient; it provides the underlying layer that secures east‑west traffic within the cluster after the request arrives.
Get Jurgen Allewijn’s stories in your inbox
Join Medium for free to get updates from this writer.
If you're managing a single AKS cluster with just a few workloads and no multi-tenant needs, AppNet might be unnecessary. For most small teams, Kubernetes `NetworkPolicy` combined with a suitable ingress covers their main requirements. Incorporating a managed mesh adds complex concepts that your team will need to learn and stay updated on. While the data plane costs are minimal and the control plane is handled by Microsoft, explaining SPIFFE identities, waypoints, ambient labels, and Gateway API semantics to a small on-call team can be a significant cognitive challenge.
Latency-critical workloads with ultra-low overhead, where every hundred microseconds matters, deserve careful benchmarking. Ambient mode is simpler than sidecar mode, but crossing a waypoint still introduces an extra hop, and HBONE tunnelling on ztunnel adds some overhead per connection. While this is negligible for most services, it may be significant for market-data fanouts or high-frequency trading bridges.
If you already have a mature Istio or Linkerd deployment and are satisfied with it, consider the migration to AppNet carefully. You should compare the migration cost with the upgrade and operational expenses needed to maintain your current setup. While the migration process is smooth, ambient mode coexists with sidecars during the transition, and it is not without costs.
Currently, AppNet assumes your clusters are on AKS. If you have a hybrid environment with some clusters in other clouds or on-premises OpenShift, AppNet won't extend to cover them. In those cases, using a cross-cloud mesh based on open-source Istio or Cilium’s cluster mesh remains the suitable solution.
Security and operational considerations
Security architects will ask three key questions (AppNet security overview). First, how is the trust domain established? In AppNet, it is linked to your AppNet resource and uses certificates issued by the managed control plane, which are automatically rotated as SPIFFE‑compliant workload certificates. The root material is not exposed to you, which security teams appreciate (no custom CA to manage), but they must understand that this means trusting Microsoft’s key management and the associated audit trail.
Second, what is the blast radius? Ztunnel operates as a DaemonSet, so if a node is compromised, only the pods on that node are affected. AppNet does not extend this impact further. A waypoint, which is a namespaced Deployment, has a scope limited to the services that use it, smaller than a cluster‑wide Istio ingress.
Third, what happens during a break‑glass scenario? If the control plane becomes unavailable, ztunnel continues enforcing the last known configuration and does not fail open. This behavior is expected and should be recorded in runbooks.
The cost model comprises three components: the AppNet resource charged per region, the data plane footprint (with ztunnel using a few hundred megabytes of memory per node and waypoints scaling with traffic), and the observability expenses you already incur for your workloads. Operationally, the biggest change is that you no longer need to run Istiod. Platform teams that previously spent a week each quarter on mesh upgrades often find that time reclaimed, while the remaining tasks mainly involve policy work, creating and reviewing AuthorizationPolicy, PeerAuthentication, Gateway, VirtualService, and Gateway API routing resources as part of regular PR reviews. This work can now be shared with your application teams, a welcome departure from the typical pattern in which only the platform team understands how the mesh operates.
A note on migrating from sidecar Istio
Teams already using Sidecar Istio can gradually switch to AppNet. The recommended approach is to deploy AppNet alongside the current Sidecar setup, enroll a non-critical namespace in ambient mode, verify that policies translate smoothly, and then migrate additional namespaces one by one. Ambient mode and sidecars can operate simultaneously; a pod with a sidecar and an ambient-enabled pod can communicate via mTLS if they share the same trust domain, and the waypoint uses the same protocols as the sidecar. The main challenge in migration is usually updating VirtualService resources that contain sidecar-specific routing tricks. These become Gateway API HTTPRoute resources on the waypoint, and while the translation process is straightforward, it can be tedious at times.
Once all namespaces have been migrated, the sidecar Istiod can be removed, returning the cluster from two coexisting control planes to a single managed control plane. Usually, this is when the platform team realizes the upgrade process has become easier.
Closing thoughts
Azure Kubernetes Application Network is not a revolution. It is a thoughtful packaging of ambient‑mode Istio into the Azure resource model, with the parts nobody enjoys operating, such as the control plane, the root CA, the upgrade cadence, and the multi‑cluster federation, lifted out of your responsibility. The pieces that are fundamentally your job, because they encode your security posture and your application topology, remain your responsibility, expressed as Kubernetes resources your team already reads every day.
For platform teams managing multiple AKS clusters and seeking an alternative to running unfettered on NetworkPolicy or incurring the sidecar costs, AppNet stands out in the Azure portfolio as a solution designed specifically for this purpose. For single-cluster setups, it might be more than necessary, and for hybrid or multi-cloud environments, it may not yet be the ideal choice. If you’re already considering adopting ambient-mode Istio, AppNet offers a managed approach. If service mesh adoption has been delayed due to operational concerns outweighing its benefits, AppNet deserves a reconsideration.
Begin with a single cluster, one namespace, and an authorization policy you've always wanted to enforce but lacked the mesh support. This initial step is smaller than any previous managed mesh on Azure, making it the ideal way to determine whether AppNet suits your platform.
The three parts collectively provide the practical AppNet overview: grasp the managed ambient architecture, implement it in real platform patterns, and determine where it integrates into your AKS estate.
Sources
Validated against the following sources in May 2026.
- Microsoft Learn — Overview of Azure Kubernetes Application Network for AKS (Preview)
- Microsoft Learn — Get Started
- Microsoft Learn — Architecture
- Microsoft Learn — Traffic Management Use Cases
- Microsoft Learn — Supported Versions
- AKS Engineering Blog — Control AI spend with per-application token rate limiting using Application Network and agentgateway
- Istio — Configure waypoint proxies
- Istio — Add workloads to the mesh
- Istio — ztunnel architecture referenc
If you found this article helpful, A (few) clap(s) 👏 or a comment 💬 really helps.
