How to Manage Dev, Staging, and Production in Kubernetes Without Duplicating YAML

Tom Jose | DevOps

11 min read

·

2 days ago

--

2

Listen

Share

A practical guide to managing dev, staging, and production with Helm, Kustomize, and GitOps

Press enter or click to view image in full size

Run this in any repository that keeps one folder of manifests per environment:

$ diff -rq k8s/staging k8s/production
Files k8s/staging/deployment.yaml and k8s/production/deployment.yaml differ
Files k8s/staging/configmap.yaml and k8s/production/configmap.yaml differ
Files k8s/staging/ingress.yaml and k8s/production/ingress.yaml differ
Only in k8s/staging: pdb.yaml
Only in k8s/staging: networkpolicy.yaml

Some of those differences are supposed to be there. Production runs more replicas and uses a different hostname. Others are accidents: a PodDisruptionBudget and a NetworkPolicy were added to staging and never copied forward. The output can’t tell you which is which, and neither can anyone reviewing the next PR.

That’s the real cost of copy-pasted YAML. Duplication isn’t the problem by itself. The problem is that intentional differences and accidental drift look identical. This article shows how to run one application across three environments with a single reusable base and small, explicit overrides, using Helm, Kustomize, and Argo CD. The goal is one application, three environments, and zero copy-pasted manifests.

Why Multi-Environment Kubernetes Becomes a YAML Nightmare

Copying manifests feels cheap on day one. You have a working deployment.yaml for dev, so you copy it to staging/ and production/, change the replica count, and move on.

The trouble starts with the second change. Someone fixes a readiness probe in production during an incident and forgets dev. Someone adds a sidecar to staging to test it and never promotes it. Six months later the three files share maybe 80% of their content, and nobody knows which 20% is deliberate. Deployments that pass in staging fail in production, because staging was never running the same thing.

Here’s the service this article deploys, checkout-api, and the only differences it should have between environments:

Configuration Development Staging Production Replicas 1 2 3 CPU request 100m 250m 500m Memory request 128Mi 256Mi 512Mi Autoscaling Disabled Enabled Enabled Secrets Dev secret store Staging secret store Production secret store Deployment Auto-sync Auto-sync Controlled sync

Everything not in that table, including probes, security context, labels, ports, and sidecars, should be identical everywhere. The repository structure exists to enforce that.

Designing a Scalable Repository Structure

The rule I build everything around is that every value lives in exactly one layer. There are four layers:

platform-charts/                     # versioned, owned by the platform team
  charts/
    service/

platform-gitops/
  apps/
    checkout-api/
      base/
        values.yaml                  # identical in every environment
      envs/
        dev/values.yaml              # only what differs in dev
        staging/values.yaml
        production/values.yaml
  clusters/
    eks-dev/cluster-values.yaml      # domain, ingress class, region
    eks-staging/cluster-values.yaml
    eks-production/cluster-values.yaml
  argocd/
    applicationsets/checkout-api.yaml
    projects/production.yaml

The shared chart defines how a service is shaped. The base defines what’s true for checkout-api everywhere. The env file defines how this service differs per environment. The cluster file defines what's true for every service in a cluster, like the ingress domain. If you're about to put a value in two places, it belongs one layer up.

One structural decision deserves an opinion: use directories for environments, not branches. Branch-per-environment (dev, staging, main) looks clean until you need it to work. Promotion becomes a merge that drags along unrelated commits, hotfixes become cherry-picks, and the difference between staging and production is hidden in Git history instead of sitting in two files you can open side by side. With directories, diff envs/staging/values.yaml envs/production/values.yaml tells you everything.

The structure also has to hold up at fifty services, not just one. Adding a service should mean adding a folder under apps/, nothing more. Cluster-wide changes, like moving every service in staging to a new ingress domain, become a one-line edit in clusters/eks-staging/cluster-values.yaml instead of fifty PRs. If a change at that scale still requires touching every service folder, a value is living in the wrong layer.

Helm vs Kustomize: Choosing the Right Approach

Both tools solve the same problem from opposite directions. Helm generates YAML from templates and values. Kustomize starts with plain YAML and patches it.

Helm wins when many services share one shape. One chart, maintained once, renders fifty services, and each service only supplies values. Here’s the base for checkout-api:

# apps/checkout-api/base/values.yaml
image:
  repository: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/checkout-api
service:
  port: 8080
probes:
  readinessPath: /ready
  livenessPath: /healthz
config:
  LOG_FORMAT: json
  PAYMENT_TIMEOUT_MS: "3000"
replicaCount: 1
resources:
  requests:
    cpu: 100m
    memory: 128Mi
  limits:
    memory: 256Mi
autoscaling:
  enabled: false
podDisruptionBudget:
  enabled: false
externalSecret:
  enabled: true
  key: checkout-api/runtime

And here’s everything production adds:

# apps/checkout-api/envs/production/values.yaml
image:
  tag: 4.12.3
resources:
  requests:
    cpu: 500m
    memory: 512Mi
  limits:
    memory: 1Gi
autoscaling:
  enabled: true
  minReplicas: 3
  maxReplicas: 12
  targetCPUUtilizationPercentage: 65
podDisruptionBudget:
  enabled: true
  minAvailable: 2
config:
  FEATURE_SPLIT_PAYMENTS: "false"

The dev file is six lines: an image tag and nothing else, because the base already holds dev’s values.

Kustomize wins when you don’t own the YAML or don’t want templating at all. The same service as a Kustomize base and overlay looks like this:

# overlays/production/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: checkout
resources:
  - ../../base
  - pdb.yaml
images:
  - name: checkout-api
    newName: 123456789012.dkr.ecr.ap-south-1.amazonaws.com/checkout-api
    newTag: 4.12.3
patches:
  - path: resources-patch.yaml
  - target:
      kind: HorizontalPodAutoscaler
      name: checkout-api
    patch: |-
      - op: replace
        path: /spec/minReplicas
        value: 3
      - op: replace
        path: /spec/maxReplicas
        value: 12

# overlays/dev/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: checkout
resources:
  - ../../base
patches:
  - patch: |-
      $patch: delete
      apiVersion: autoscaling/v2
      kind: HorizontalPodAutoscaler
      metadata:
        name: checkout-api

My rule of thumb is simple. Use Helm for your own services, because a shared chart is how you stop fifty teams writing fifty Deployments. Use Kustomize for third-party manifests you want to patch without forking, and as a last-mile layer on top of Helm when one environment needs something the chart doesn’t expose. Argo CD can render a Helm chart through Kustomize’s helmCharts field (with --enable-helm), but if you reach for that combination often, your chart is missing a value.

The case where combining them makes sense is vendor charts. Say the upstream chart for a tool like the AWS Load Balancer Controller doesn’t expose a topologySpreadConstraints value, and production needs one. Forking the chart means owning every future upgrade. A Kustomize patch over the rendered chart is ten lines, survives upgrades, and lives in the production overlay where reviewers expect environment-specific changes.

Managing Environment-Specific Configuration

Most of the table maps to values directly. Replicas and autoscaling hide the first sharp edge.

Production has replicaCount: 3 and an HPA. If the chart always renders spec.replicas, Argo CD and the HPA fight. The HPA scales to 8 during a traffic spike, Argo CD sees drift against Git's 3, and selfHeal scales it back down. That's 5 Pods gone at peak load, triggered by your deployment tool. With autoscaling on, the replica count belongs to the HPA, and minReplicas: 3 is how production expresses "three replicas."

The second sharp edge is conditional logic. Wrong:

# templates/deployment.yaml
spec:
{{- if eq .Values.environment "production" }}
  replicas: 3
{{- else if eq .Values.environment "staging" }}
  replicas: 2
{{- else }}
  replicas: 1
{{- end }}

Once a chart knows environment names, every new environment means editing templates, and nobody can tell what production gets without reading Go templates. Right:

# templates/deployment.yaml
spec:
{{- if not .Values.autoscaling.enabled }}
  replicas: {{ .Values.replicaCount }}
{{- end }}
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/configmap.yaml") . | sha256sum }}

The chart asks about capabilities (autoscaling.enabled, podDisruptionBudget.enabled), never about which environment it's in. The values files decide which capabilities each environment turns on. The checksum/config annotation solves a quieter problem: changing a ConfigMap value doesn't restart Pods on its own, so without it, a config change reaches production in Git while the Pods keep running the old config. Kustomize avoids this with configMapGenerator, which hashes the ConfigMap name.

Hostnames come from the cluster layer, not the app:

# clusters/eks-production/cluster-values.yaml
ingress:
  className: alb
  domain: payflow.io
region: ap-south-1

The chart builds checkout.payflow.io from the service name and domain, so no service file ever hardcodes a hostname. Feature flags work for a handful of on/off settings in config. Once you're toggling them per customer or rolling out gradually, move them to a feature-flag service, because a Git commit plus a rollout is a slow kill switch.

GitOps for Multi-Environment Deployments

One ApplicationSet deploys all three environments, layering the values files in order from least to most specific (later files win):

apiVersion: argoproj.io/v1alpha1
kind: ApplicationSet
metadata:
  name: checkout-api
  namespace: argocd
spec:
  goTemplate: true
  goTemplateOptions: ["missingkey=error"]
  generators:
    - list:
        elements:
          - env: dev
            cluster: eks-dev
            chartVersion: service-chart-2.5.0
            autoSync: "true"
          - env: staging
            cluster: eks-staging
            chartVersion: service-chart-2.4.1
            autoSync: "true"
          - env: production
            cluster: eks-production
            chartVersion: service-chart-2.4.1
            autoSync: "false"
  template:
    metadata:
      name: 'checkout-api-{{.env}}'
    spec:
      project: '{{.env}}'
      sources:
        - repoURL: https://github.com/payflow-eng/platform-charts.git
          targetRevision: '{{.chartVersion}}'
          path: charts/service
          helm:
            valueFiles:
              - $values/apps/checkout-api/base/values.yaml
              - $values/clusters/{{.cluster}}/cluster-values.yaml
              - $values/apps/checkout-api/envs/{{.env}}/values.yaml
        - repoURL: https://github.com/payflow-eng/platform-gitops.git
          targetRevision: main
          ref: values
      destination:
        name: '{{.cluster}}'
        namespace: checkout
      syncPolicy:
        syncOptions:
          - CreateNamespace=true
  templatePatch: |
    {{- if eq .autoSync "true" }}
    spec:
      syncPolicy:
        automated:
          prune: true
          selfHeal: true
    {{- end }}

Dev and staging sync automatically. Production shows OutOfSync after a merge and waits for someone to press Sync. The chart version is pinned per environment, so a new chart release goes through dev first, exactly like an application release.

Promotion moves an artifact, not a rebuild. CI builds checkout-api once, tags the image with an immutable version, and commits that tag to envs/dev/values.yaml. Promoting is a PR that copies the tag forward:

TAG=$(yq '.image.tag' apps/checkout-api/envs/staging/values.yaml)
yq -i ".image.tag = \"${TAG}\"" apps/checkout-api/envs/production/values.yaml
git switch -c "promote/checkout-api-${TAG}"
git commit -am "Promote checkout-api ${TAG} to production"
gh pr create \
  --title "Promote checkout-api ${TAG} to production" \
  --body "Image verified in staging. Diff limited to image.tag." \
  --reviewer payflow-eng/sre

Rollback uses the same mechanism in reverse. Revert the promotion PR, and production shows OutOfSync against the previous tag, ready to sync. There's no separate rollback procedure to forget during an incident, and the revert sits in Git history next to the change it undid.

Now for preventing accidental production deployments. CODEOWNERS with branch protection requires SRE approval for anything under apps/*/envs/production/ and clusters/eks-production/. The production AppProject only allows destinations in eks-production, so a mistyped cluster name in a dev element can't land there. A sync window blocks production syncs over the weekend:

apiVersion: argoproj.io/v1alpha1
kind: AppProject
metadata:
  name: production
  namespace: argocd
spec:
  sourceRepos:
    - https://github.com/payflow-eng/platform-charts.git
    - https://github.com/payflow-eng/platform-gitops.git
  destinations:
    - name: eks-production
      namespace: checkout
  syncWindows:
    - kind: deny
      schedule: "0 18 * * 5"
      duration: 62h
      timeZone: Asia/Kolkata
      applications: ["*"]

The sharp edge that bites layered setups is the base file. Promotion copies an image tag, but a change to base/values.yaml reaches every environment in the same merge. Someone bumps PAYMENT_TIMEOUT_MS "for dev," and production gets it on the next sync along with whatever was actually being promoted. Treat base edits as production changes: put apps/*/base/ under the same CODEOWNERS rule, and when a change needs to be staged, make it in the env files first and move it into base once every environment agrees.

Managing Secrets Across Environments

No secret values go in Git, encrypted or otherwise. The chart renders an ExternalSecret, and External Secrets Operator pulls the value from AWS Secrets Manager:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: checkout-api
  namespace: checkout
spec:
  refreshInterval: 1h
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-secrets
  target:
    name: checkout-api-secrets
  dataFrom:
    - extract:
        key: checkout-api/runtime

Notice the store is called aws-secrets in every environment, and the key carries no environment prefix. Each cluster has its own ClusterSecretStore with that name, and the ESO controller in each cluster authenticates through EKS Pod Identity to a different AWS account (one per environment):

apiVersion: external-secrets.io/v1
kind: ClusterSecretStore
metadata:
  name: aws-secrets
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-south-1

That’s how the “Dev / Staging / Production secret store” rows in the table get enforced: through IAM and account boundaries, not naming. The alternative I often see is one shared account with keys like staging/checkout-api/runtime and production/checkout-api/runtime. In that setup, a one-word typo in a staging values file hands production database credentials to a staging Pod, and nothing stops it. With separate accounts, the staging cluster's role can't read production secrets, whatever the YAML says. Vault works the same way, with a separate auth role and policy path per cluster.

There’s one more sharp edge, and it shows up during the first credential rotation. ESO refreshes the Kubernetes Secret on its refreshInterval, but Pods that read the secret as environment variables keep the old value until they restart. The database rotates its password, the Secret updates, and every running checkout-api Pod keeps using a password that no longer works. Either mount secrets as files and have the application re-read them, or use a controller like Stakater Reloader (secret.reloader.stakater.com/reload: "checkout-api-secrets" on the Deployment) to roll Pods when the Secret changes. Test rotation in staging before production does it for you.

Preventing Configuration Drift and Deployment Failures

Layered values have one real weakness: reviewers read the input, but the cluster runs the output. A five-line values diff can produce a 200-line manifest change, or silently produce nothing because of a typo in a key name. So CI renders every environment on every PR and validates what comes out:

#!/usr/bin/env bash
set -euo pipefail

for env in dev staging production; do
  helm template checkout-api ./platform-charts/charts/service \
    -f apps/checkout-api/base/values.yaml \
    -f "clusters/eks-${env}/cluster-values.yaml" \
    -f "apps/checkout-api/envs/${env}/values.yaml" \
    > "rendered/${env}.yaml"

  kubeconform -strict -summary \
    -kubernetes-version 1.34.0 \
    -schema-location default \
    -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' \
    "rendered/${env}.yaml"

  kyverno apply ./policies --resource "rendered/${env}.yaml"
done

The same job renders the target branch and posts the diff of rendered manifests per environment as a PR comment. That diff is the thing reviewers should read. A shared values.schema.json in the chart catches the typo case: with additionalProperties: false, autoscalling.enabled: true fails helm template instead of being silently ignored.

Kyverno policies run twice, in CI against rendered output and at admission time in the cluster, so a kubectl apply from someone's laptop gets the same rules as a PR.

Drift detection is where Argo CD earns its place. In dev and staging, selfHeal reverts manual changes within minutes. In production, where sync is manual, drift shows up as OutOfSync, so alert on it before it becomes the new normal:

- alert: ProductionAppOutOfSync
  expr: argocd_app_info{project="production", sync_status="OutOfSync"} == 1
  for: 2h
  labels:
    severity: warning
  annotations:
    summary: "{{ $labels.name }} has been out of sync for 2h: unsynced merge or manual change in the cluster"

Multi-environment checklist

• One base values file per service; env files contain only differences
• Environments are directories, not branches
• Hostnames, domains, and regions live in cluster-level values
• Charts check capabilities (autoscaling.enabled), never environment names
• spec.replicas is omitted whenever an HPA is enabled
• Config changes trigger rollouts through a checksum annotation or configMapGenerator
• Images are built once and promoted by tag
• Chart versions are pinned per environment
• CODEOWNERS covers production env files and base files
• Production syncs manually inside allowed sync windows
• Each environment reads secrets from its own account or store
• CI renders, validates, and diffs every environment on every PR
• Production OutOfSync state alerts after a set time

Fewer files is a side effect. The real outcome is that every difference between your environments becomes a decision someone made on purpose, in a file small enough to read in ten seconds. When production breaks and staging didn’t, the investigation should start and mostly end with diff envs/staging/values.yaml envs/production/values.yaml. If that command can't explain the difference, your structure is still hiding something.

Tom Jose is a DevOps Engineer at KotaiCode, where the team helps companies get their Kubernetes and AWS infrastructure production-ready.