Why Kubernetes Namespaces Aren’t a Real Security Boundary

Tom Jose | DevOps

10 min read

·

18 hours ago

--

Listen

Share

Shared kernels, RBAC limitations, resource isolation, and how to build secure multi-tenant Kubernetes clusters

Press enter or click to view image in full size

Kubernetes namespaces were never designed to be a security boundary. They were designed so two teams could both name a Deployment api without colliding. Somewhere between that design and today's shared platforms, "each team gets a namespace" turned into "each team is isolated," and those are very different promises.

A namespace scopes names and API permissions. It doesn’t scope the kernel your containers run on, the node they share, the network they talk over, or the CPU and memory they compete for. If one tenant’s workload is compromised, the namespace alone stops almost nothing.

This article walks through one scenario. Two teams share a production EKS cluster, one team’s Pod gets compromised, and each section adds the control that closes the next path the attacker would take. By the end, you’ll have a layered model for multi-tenancy and a clear sense of when a namespace is enough and when it isn’t.

The Scenario

The cluster is eks-prod-shared. The payments team runs payments-api and stores database credentials in a Kubernetes Secret. The growth team runs image-resizer, a service built on a third-party image library. Each team has its own namespace and its own RoleBindings. Leadership believes the teams are isolated.

On a Tuesday, a remote code execution bug in the image library is disclosed. Before growth patches it, an attacker uploads a crafted image and gets a shell inside an image-resizer Pod. From there, here's what a namespace-only cluster allows.

What Kubernetes Namespaces Actually Isolate

Namespaces partition namespaced API objects: Pods, Services, Secrets, ConfigMaps, Roles, and similar. A RoleBinding in growth grants nothing in payments. That part works.

Everything else is shared by every tenant:

$ kubectl api-resources --namespaced=false -o name | head -12
apiservices.apiregistration.k8s.io
certificatesigningrequests.certificates.k8s.io
clusterroles.rbac.authorization.k8s.io
customresourcedefinitions.apiextensions.k8s.io
ingressclasses.networking.k8s.io
mutatingwebhookconfigurations.admissionregistration.k8s.io
namespaces
nodes
persistentvolumes
priorityclasses.scheduling.k8s.io
storageclasses.storage.k8s.io
validatingwebhookconfigurations.admissionregistration.k8s.io

Nodes, PersistentVolumes, CRDs, PriorityClasses, and admission webhooks are all cluster-scoped. So are the things that aren’t API objects at all: the kernel on each node, the Pod network, cluster DNS, and the node’s IAM role. Every Service in the cluster also resolves from every namespace. Our attacker can run nslookup payments-api.payments.svc.cluster.local and get a valid IP back on the first try.

Why RBAC Alone Isn’t Enough

The attacker’s first move is to check what the Pod’s identity can do. Kubernetes mounts a ServiceAccount token into every Pod by default:

$ kubectl auth can-i --list --as=system:serviceaccount:growth:image-resizer -n growth
Resources                Verbs
pods                     [create delete get list patch update watch]
pods/exec                [create]
deployments.apps         [create delete get list patch update watch]
secrets                  [get list watch]
...

Someone bound image-resizer to the built-in edit ClusterRole months ago, so the service could restart its own workers. RBAC is doing exactly what it was told, and none of these permissions reach payments. The problem is that create pods is far more powerful than it looks.

Anyone who can create Pods in a namespace can mount every Secret in that namespace and run as every ServiceAccount in it, so they inherit the highest privilege available there. They can also choose what that Pod looks like: privileged, with the host filesystem mounted, scheduled onto whichever node they like. RBAC controls which API calls you can make. It has no opinion on what the Pods you create are allowed to do once they’re running. That’s the job of admission control, covered below.

The fix starts with not handing out identity by default. Wrong:

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: image-resizer-edit
  namespace: growth
subjects:
  - kind: ServiceAccount
    name: image-resizer
    namespace: growth
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: edit

Right:

apiVersion: v1
kind: ServiceAccount
metadata:
  name: image-resizer
  namespace: growth
automountServiceAccountToken: false

Most application Pods never call the Kubernetes API, so they shouldn’t carry a token at all. The few that do should get a Role with exactly the verbs and resource names they need. Also audit for the RBAC verbs that are effectively cluster-admin in disguise: escalate, bind, impersonate, nodes/proxy, and write access to webhook configurations.

The Shared Kernel Problem

Suppose RBAC is tight, but the attacker still has a shell. The next target is the node.

Containers are processes with namespaces and cgroups applied. Every Pod on a node shares that node’s Linux kernel. A kernel bug or a container runtime bug is a path from one tenant’s container to the host, and from the host to every other Pod on it. This isn’t theoretical. CVE-2024–21626 in runc let a crafted container reach the host filesystem through a leaked file descriptor.

Once on the host, the attacker can read every Secret volume the kubelet has mounted for other Pods on that node, under /var/lib/kubelet/pods/. If a payments-api replica happens to be scheduled next to image-resizer, its database credentials are sitting on that disk. Kubernetes' Node authorizer limits a kubelet to the Secrets of Pods scheduled on its own node, which is exactly the point: whoever owns the node owns everything scheduled on it.

On EKS there’s one more prize on the node: its IAM role. If a Pod can reach the instance metadata service, it can pull the node role’s credentials, which typically include ECR pull, EC2 describe calls, and whatever else someone attached “temporarily.” Require IMDSv2 with a hop limit of 1 on every node, so Pods can’t reach it. Karpenter’s EC2NodeClass does this by default. Managed node groups and custom launch templates often allow a hop limit of 2, so check them. Use EKS Pod Identity for workloads that need AWS access.

Two kernel-level controls reduce the blast radius. User namespaces, GA in Kubernetes 1.36, map root inside the container to an unprivileged UID on the host, so a breakout lands as a nobody:

spec:
  hostUsers: false

Seccomp profiles (RuntimeDefault at minimum) cut the number of syscalls an attacker can use to reach kernel bugs. Neither removes the shared kernel. Both make exploiting it much harder.

Preventing Privileged Workloads

Without a kernel bug, the easiest path to the host is to ask for it. With create pods permission and no admission controls, the attacker can simply submit this:

apiVersion: v1
kind: Pod
metadata:
  name: resizer-debug
  namespace: growth
spec:
  hostPID: true
  containers:
    - name: debug
      image: public.ecr.aws/docker/library/busybox:1.36
      command: ["sleep", "infinity"]
      securityContext:
        privileged: true
      volumeMounts:
        - name: host
          mountPath: /host
  volumes:
    - name: host
      hostPath:
        path: /

That’s full node access, granted by a namespace-scoped permission. Pod Security Admission blocks it with three labels on the namespace:

apiVersion: v1
kind: Namespace
metadata:
  name: growth
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/enforce-version: latest
    pod-security.kubernetes.io/warn: restricted

The restricted profile rejects privileged containers, host namespaces, hostPath volumes, root users, added capabilities, and missing seccomp profiles. Here's the sharp edge: PSA only applies to namespaces that carry the label. A tenant who can create namespaces, or a platform template that forgets the label, quietly falls back to no enforcement. Set a cluster-wide default through the AdmissionConfiguration where you control the API server. On EKS, where you don't, enforce the label itself with a policy that rejects namespaces created without it.

PSA covers the Pod spec. Tenant-specific rules need a policy engine. ValidatingAdmissionPolicy (GA since 1.30) handles them in CEL without running a webhook, and the next section uses it.

Resource Isolation: ResourceQuota and LimitRange

The attacker doesn’t need secrets to hurt payments. They only need the node. A Deployment of 200 replicas with no limits, a Pod that fills the node's disk with temp files, or a fork bomb can starve or evict payments-api Pods sharing the same nodes. Availability is a security property too.

ResourceQuota caps what a namespace can consume. LimitRange gives every container sensible defaults so the quota actually applies:

apiVersion: v1
kind: ResourceQuota
metadata:
  name: growth-quota
  namespace: growth
spec:
  hard:
    requests.cpu: "40"
    requests.memory: 80Gi
    limits.memory: 120Gi
    requests.ephemeral-storage: 100Gi
    limits.ephemeral-storage: 200Gi
    pods: "150"
    services.loadbalancers: "2"
---
apiVersion: v1
kind: LimitRange
metadata:
  name: growth-defaults
  namespace: growth
spec:
  limits:
    - type: Container
      default:
        memory: 512Mi
        ephemeral-storage: 2Gi
      defaultRequest:
        cpu: 100m
        memory: 256Mi
        ephemeral-storage: 512Mi
      max:
        memory: 8Gi

Watch two gaps. First, PriorityClasses are cluster-scoped. A tenant who sets priorityClassName: system-cluster-critical can preempt other tenants' Pods, so add a quota with a scopeSelector on PriorityClass that limits each namespace to the classes you've approved. Second, quotas don't cover process IDs. Set the kubelet's podPidsLimit so one Pod's fork bomb can't exhaust PIDs for the whole node.

Network Isolation Between Namespaces

The Kubernetes network is flat by default. From image-resizer, the attacker can reach payments-api, the payments database proxy, and every other Pod IP in the cluster. Namespaces have no effect on routing at all.

Start every tenant namespace with default deny in both directions, then add back only what’s needed. DNS is the one everyone forgets:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: default-deny
  namespace: growth
spec:
  podSelector: {}
  policyTypes: [Ingress, Egress]
---
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-dns-and-internet
  namespace: growth
spec:
  podSelector: {}
  policyTypes: [Egress]
  egress:
    - to:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: kube-system
          podSelector:
            matchLabels:
              k8s-app: kube-dns
      ports:
        - { protocol: UDP, port: 53 }
        - { protocol: TCP, port: 53 }
    - to:
        - ipBlock:
            cidr: 0.0.0.0/0
            except:
              - 10.20.0.0/16        # cluster VPC: no lateral movement
              - 169.254.169.254/32  # instance metadata
      ports:
        - { protocol: TCP, port: 443 }

The sharp edge is enforcement. NetworkPolicy objects are only enforced if your CNI enforces them. On EKS, the VPC CNI needs network policy enabled explicitly, or you install Cilium or Calico. A cluster without enforcement accepts these objects, stores them, and silently ignores them. Test it: run a Pod in growth and try to curl a Service in payments. If it connects, your policies are decoration.

Stronger Isolation Strategies

Everything so far still shares nodes between tenants. For tenants with different trust levels, that’s the remaining hole: one kernel exploit beats every control above. There are three ways to stop sharing.

Dedicated node pools give each tenant its own nodes. With Karpenter, a NodePool per tenant carries a taint and a label:

apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: tenant-growth
spec:
  template:
    metadata:
      labels:
        tenant.payflow.io/team: growth
    spec:
      taints:
        - key: tenant.payflow.io/team
          value: growth
          effect: NoSchedule
      nodeClassRef:
        group: karpenter.k8s.aws
        kind: EC2NodeClass
        name: default

Here’s the sharp edge that defeats most “dedicated node” setups: taints aren’t a security control. Any Pod can add a toleration for any taint, and any Pod can set a nodeSelector for any label. A compromised growth workload can schedule itself onto payments nodes. Enforce placement in admission instead:

apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicy
metadata:
  name: tenant-node-placement
spec:
  failurePolicy: Fail
  matchConstraints:
    resourceRules:
      - apiGroups: [""]
        apiVersions: ["v1"]
        operations: ["CREATE", "UPDATE"]
        resources: ["pods"]
  validations:
    - expression: >-
        has(object.spec.nodeSelector) &&
        'tenant.payflow.io/team' in object.spec.nodeSelector &&
        object.spec.nodeSelector['tenant.payflow.io/team'] ==
        namespaceObject.metadata.labels['tenant.payflow.io/team']
      message: "Pods must target their own tenant's node pool."
---
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingAdmissionPolicyBinding
metadata:
  name: tenant-node-placement
spec:
  policyName: tenant-node-placement
  validationActions: [Deny]
  matchResources:
    namespaceSelector:
      matchExpressions:
        - key: tenant.payflow.io/team
          operator: Exists

Sandboxed runtimes keep shared nodes but give each Pod its own kernel. Kata Containers runs every Pod in a lightweight VM, and gVisor intercepts syscalls in a user-space kernel. Both are selected per Pod with a RuntimeClass:

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: kata
handler: kata

The cost is real. Kata needs hardware virtualization (on EC2, usually metal instances), and gVisor adds syscall overhead that hurts I/O-heavy workloads. Use sandboxes for untrusted code, like customer-supplied plugins or CI build jobs, not as a default for everything.

Separate clusters are the only option where the control plane, CRDs, admission webhooks, and cluster-scoped resources aren’t shared at all. On EKS, add separate AWS accounts and the IAM boundary comes along for free.

Designing Secure Multi-Tenant Kubernetes

Here’s how the three common architectures compare:

Press enter or click to view image in full size

My default is a hybrid. Internal teams with similar trust share a cluster with namespaces plus every control above. Workloads that handle regulated data get dedicated node pools enforced by admission. Untrusted code runs in a sandboxed runtime. Anything owned by an external customer gets its own cluster. Virtual clusters (vCluster and similar) sit in the middle and give each tenant its own API server and CRDs, but they still share nodes, so the kernel question remains.

Back to the attacker in image-resizer, with every layer applied:

Press enter or click to view image in full size

Multi-tenant cluster checklist

• Application ServiceAccounts don’t automount tokens
• No tenant binding to edit or admin for workloads; audit escalate, bind, impersonate
• Every tenant namespace enforces Pod Security restricted, and unlabeled namespaces are rejected
• Seccomp RuntimeDefault everywhere; hostUsers: false where the runtime supports it
• IMDSv2 required with hop limit 1 on every node
• ResourceQuota and LimitRange in every tenant namespace, including ephemeral storage
• PriorityClass usage limited per namespace; kubelet podPidsLimit set
• Default-deny NetworkPolicy in both directions, enforcement verified with a real test
• Tenant node placement enforced by admission, not just taints
• Untrusted code runs in a sandboxed runtime or a separate cluster

A namespace answers “whose is this?” Security questions are about “what can this reach?”, and the answer to that is decided by the kernel, the network, the node, and the admission chain, none of which read namespace boundaries on their own. Multi-tenancy in Kubernetes is a stack of controls you assemble yourself, and the right height for that stack depends on how much you trust the tenant on the other side. Decide that first, and the architecture mostly chooses itself.

Tom Jose is a DevOps Engineer at KotaiCode, where the team helps companies get their Kubernetes and AWS infrastructure production-ready.