Sitemap
Open in app

Sign in

Medium Logo
Write
Search

Sign in

Hudeyfa Jama

Hudeyfa Jama

Engineer Substack - https://blog.thepattern.dev

How I’d Secure a GKE Platform End to End

Hudeyfa Jama
16 min read3 days ago

--

Introduction

Asalaamu Aleykum, my name is Hudeyfa. For five years, my job has been making platforms on Google Cloud faster and easier to use, and over the last couple of years I’ve watched the number of cyber attacks keep increasing with AI at the centre of it all.

This month has already earned a special name: Black October. In the space of 48 hours, we’ve had several incidents that in a typical year would be among the biggest incidents of the entire year.

And this is just the beginning. Most cybersecurity experts reckon that it’s only going to get worse before it starts getting better.

That’s what really got me interested in cybersecurity.

I wanted to understand how you should actually think about security end to end, from the code to the cloud.

Like most good explanations, I think this one is best told through a story: an imaginary attack (also because a real attack would be, well, illegal).

Okay, but first of all buckle up and grab a cup of tea, because this is going to be on the longer side, but hopefully one you can keep coming back to :).

In a hurry? Skip straight to the top ten actions.

Scenario

Talha is a developer on a team at Northwind that runs a customer-facing API on GKE. On a Friday afternoon, he ships a small feature: a new endpoint that lets customers export their reports. The pull request passes review, the tests go green and the pipeline deploys to production before Talha has finished his chai.

The export feature pulls in a library and that library pulls in others. Three levels down sits a package with a remote code execution bug.

Press enter or click to view image in full size

On Saturday, the package’s maintainers release a fix and publish a security advisory describing the bug. The thing is, attackers read advisories too. Within minutes, powered by our good friend Claude, they start scanning the internet for anything still running the vulnerable version and unfortunately for us, Talha’s new endpoint answers.

Press enter or click to view image in full size

The endpoint takes a report template from the request and passes it to the library to render. The attacker sends a template with a payload hidden inside it where the report fields should be. The vulnerable package evaluates that payload as code and it runs with the same permissions as Talha’s service. No login and no stolen password are needed. One crafted request and an attacker has code running inside Talha’s container.

The container runs as root, with a shell and curl still in the image, because the Dockerfile was copied from another service years ago:

FROM python:3.12
# Installed once to debug something, never removed
RUN apt-get update && apt-get install -y curlWORKDIR /app
COPY . .
RUN pip install -r requirements.txt# No USER instruction, so the app runs as root
CMD ["gunicorn", "app:app", "--bind", "0.0.0.0:8080"]

The attacker asks the GKE metadata server for credentials and gets a token for the workload’s Google service account. That account was given a broad role “temporarily” two years ago, by an engineer who has since left.

The Wiz runtime sensor notices. It raises a high-severity alert within a second, but there is no Runtime Response Policy configured to terminate it. This is a problem as nobody is watching the queue on a Saturday morning.

Press enter or click to view image in full size

With the access they’ve gained, they now list storage buckets across three GCP projects, read two secrets and copy a customer export to a bucket they own. On Monday morning, Talha gets a message from the security team asking why his service was reading buckets in a project it has never used.

Press enter or click to view image in full size
Press enter or click to view image in full size

Talha didn’t do anything wrong per se. Each layer just didn’t follow best practices and those security holes all lined up to result in the breach.

But what are those layers, and how can we conceptualise them?

The 4Cs model

Cloud. Cluster. Container. Code. Long ago the four layers of the cloud lived in harmony… until the code layer was hacked. Only the security engineer, master of all Cs, could stop them.

Press enter or click to view image in full size

Rewind Talha’s breach and we can see that it touches all four layers:

  • Code — the bug was in the code
  • Container — the root shell came with the container
  • Cluster — handed over a token
  • Cloud — excessive cross-project IAM permissions

The 4Cs model, also known as the cloud-native security model, was a term coined by the open source Kubernetes community as part of establishing a Defence in Depth (DiD) inspired framework specifically designed for containerised applications.

The more in vogue framework for thinking about containerised application security is to organise security around the CNCF’s lifecycle phases instead: develop, distribute, deploy and runtime.

I still find the 4Cs a clear and useful way to split responsibility on a platform team but it doesn’t come without flaws and I will touch on its shortcomings later.

Shared responsibility

A question that comes to a lot of developers’ minds: doesn’t Google (or your favourite hyperscaler) own the infra? And if so, shouldn’t they be responsible for the whole thing? Right? Right?

The shared responsibility model (introduced by AWS in 2011) details what the cloud providers are responsible for and what you, the customer, are responsible for. Essentially in our GKE scenario, Google runs the control plane and the infrastructure beneath it. We own pretty much everything else on top of that: IAM, networking, workloads, images, code and data, plus node configuration. Crucially, we also own the monitoring and the response which was missing in our scenario.

So yes, we are partially responsible for it.

Press enter or click to view image in full size

Code

Press enter or click to view image in full size

Everything started from that pesky code execution bug in a vulnerable dependency so let’s start there. That’s now one of the most common ways into a GCP org: exploited third-party software accounted for 44.5% of initial access in the second half of 2025, according to Google’s Threat Horizons report.

Going back to our scenario on that Friday, quilltemplate had no known vulnerabilities, so no scanner would have picked up and blocked Talha’s pull request. The CVE was published on Saturday and exploited 17 minutes later.

Press enter or click to view image in full size

So realistically, the only two things at the code level would have stopped this attack:

  • A code design change — Talha’s endpoint accepted a template from the user and ran it. This is a classic bug called server-side template injection (SSTI). A code reviewer asking the right question should have caught this and I do think an AI powered review would have as well.
  • Security knowledge — An API test in staging that sends template syntax to /export would have caught the same flaw (See OWASP's testing guide).

Both the test and the review depend on developers knowing SSTI is a risk in the first place, which is why the NCSC lists keeping security knowledge sharp as a core principle of secure development. But humans and AI are not infallible so if not preventing how could we have improved the response time?

  • Scheduled dependency scanning (SCA) — A scheduled scan would have flagged it on Saturday morning. Dependabot did open a pull request on Saturday, but nobody saw it until Monday.
  • Patching on a clock — Set fix deadlines for exploitable critical vulnerabilities in hours, with Renovate or Dependabot opening the upgrade pull request automatically and paging the on-call engineer to approve it the same day.
  • A WAF at the edge — Cloud Armor’s preconfigured OWASP rules in front of the load balancer block common injection patterns, and rate limiting slows the scan that found the endpoint. It wouldn’t reliably catch a brand-new payload, but it buys time.

Wider code risks

  • Software Bill of Materials (SBOM) — Keep an inventory of every third-party component. The UK’s National Cyber Security Centre asks for this in its Software Security Code of Practice, with an SBOM as one way to do it. (Wondering how that differs from SCA? SCA actively hunts for problems; an SBOM is a static list you check when a new CVE lands.)
  • Brand-new packages — Commit lockfiles and hold new package versions back for a few days before CI can pull them. Most npm worms are caught within that window.
  • Secrets in code — Secret scanning with push protection and secrets held in Secret Manager.
  • Your pipeline — The OWASP Top 10 CI/CD Security Risks is the checklist here. The essentials:
  • Protect the main branch, require reviews and use CODEOWNERS on pipeline files so nobody quietly edits the deploy step
  • Pin third-party actions (e.g. GitHub Actions) by commit SHA, use ephemeral runners and give build jobs only the permissions they need
  • Never run workflows from forks or outside pull requests with access to secrets aka PPE (poisoned pipeline execution)
  • Authenticate to Google Cloud with OIDC through Workload Identity Federation
  • AI review — AI-powered review tools have been around for a while, and Google’s recently released one (CodeMender) would likely have caught and fixed this one autonomously.

Trade-offs

Every security gate we implement introduces some friction, and where there is friction there is a desire to take the path of least resistance and bypass it.

  • Blocking deployments on low-medium CVEs -> results in force deploying in all cases (including high)
  • Security scans taking too long on each build -> results in those jobs being skipped or worse disabled
  • Rubber-stamp approval carried out by a CODEOWNER who is missing context -> results in code being approved that could have major flaws

Container

Press enter or click to view image in full size

The second failing was an image packed with tools it didn’t need.

Ironically, curl gave the attacker away. A hardened image would have forced them to use Python, which looks like normal app traffic and is much harder to spot.

But we don’t want irony to be our defence, our defence should be ironclad. So here is what we can do at the container level:

  • A minimal, non-root image. Build on distroless, Chainguard or Docker Hardened Images, and use a multi-stage build so compilers and package managers never reach the final image. Distroless is Google’s own project, so it’s the native choice on GKE.
  • A locked-down runtime. Two Linux features box an attacker in once they’re inside.
  • Seccomp limits which system calls the process can make, so it can read files but never spawn a shell.
  • AppArmor limits which files, devices and network resources it can reach.
Press enter or click to view image in full size

Wider container risks

Everything above is about the one image in the story. These nine habits keep every image you ship honest:

Press enter or click to view image in full size

And four risks the cards don’t cover:

  • Access to the container runtime — Never mount the containerd or Docker socket into a pod, because whoever can talk to the runtime controls every container on the node.
  • Privileged containers — privileged: true switches off the isolation entirely. The container gets every capability and every host device, so a compromise inside it becomes a compromise of the node and every pod on it. Pod Security Admission at restricted blocks it.
  • Untrusted workloads — GKE Sandbox (gVisor) puts a second kernel between the container and the node.
  • GKE Autopilot — If you’d rather not configure the runtime controls yourself, Autopilot has several baked in. It blocks privileged containers and host path mounts, applies the RuntimeDefault seccomp profile, and runs every node on Container-Optimized OS with containerd.

Cluster

Press enter or click to view image in full size

The third failing was a cluster that let it all happen. Nothing stopped a root pod from being deployed, and once the attacker was inside, the cluster did exactly what it was built to do: it handed them a token for the pod’s Google identity.

Press enter or click to view image in full size

A GKE cluster is two things at once: a Google Cloud resource and a Kubernetes API. The job at this layer is to make sure a compromised pod stays a compromised pod. So here’s what we can do at the cluster level:

  • One identity per workload. Northwind already used Workload Identity, so the pod had its own Google service account. The problem was the role behind it, which the Cloud section covers.
  • Unprivileged pods, enforced. Pod Security Admission at restricted would have rejected Talha's root pod at deploy time. It wouldn't have stopped the RCE or the token request, but it limits what an attacker can do on the node.
  • No Kubernetes token it doesn’t need. The attacker took the pod’s Kubernetes token too. automountServiceAccountToken: false on pods that never call the API, and minimal RBAC for the ones that do.
Press enter or click to view image in full size

Security Command Center’s attack path simulation maps these doors before an attack. Google SecOps shows you which ones were walked through after one.

Wider cluster risks

Who can reach the cluster. Keep the control plane behind a DNS-based endpoint protected by IAM, with RBAC through Google Groups and no standing cluster-admin. Give nodes no external IPs, and run them on a minimal custom service account rather than the Compute Engine default.

What runs inside it. Pod Security Admission only covers the security context, so Policy Controller, Gatekeeper or Kyverno enforces the rest: digests, no mutable tags, resource limits. Default-deny NetworkPolicy in every namespace on Dataplane V2. Kubernetes Secrets encrypted in etcd with Cloud KMS, or the Secret Manager add-on so pods never hold one at all.

Get Hudeyfa Jama’s stories in your inbox

Join Medium for free to get updates from this writer.

Keeping it current. A release channel with auto-upgrade and someone reading the GKE security bulletins. Shielded GKE Nodes with secure boot turned on. And since ingress-nginx was retired in March 2026, move to Gateway API.

Cloud

Press enter or click to view image in full size

The fourth and final failing was the cloud itself. One forgotten role turned a single compromised pod into access across three projects, and nothing at the organisation level stopped the data leaving.

Identity is the new perimeter.

Press enter or click to view image in full size

Almost every serious breach path runs through IAM. Networks still matter, but a valid credential walks straight past a firewall. So this layer starts with identity, then guardrails, then data. Here’s what would have stopped it:

  • Least privilege you actually maintain. Scoped to its own bucket, the token couldn’t have reached finance-data-prod or billing-prod. IAM Conditions make “temporary” access expire, and IAM Recommender strips what nobody uses.
  • A perimeter around sensitive data. VPC Service Controls around the customer and finance projects would have blocked the copy to the attacker’s bucket, even with a valid token.
  • Controlled egress. The perimeter only covers Google’s APIs. Once it blocks the copy, the attacker’s next move is to read the export through the pod and send it out over HTTPS, or open a reverse shell. Secure Web Proxy with an allowlist of the destinations the app needs stops both.

Wider cloud risks

The Cloud Security Alliance’s Top Threats to Cloud Computing 2024 ranks these three highest. On Google Cloud they look like this:

  1. Misconfiguration — Organization Policies that block public buckets, external sharing and external IPs by default, so one wrong click can’t open you up.
  2. Identity and access management — Phishing-resistant MFA, no service account keys and Privileged Access Manager for just-in-time admin access.
  3. Insecure interfaces and APIs — Talha’s /export endpoint was one. Authenticate every public endpoint and keep Cloud Armor in front of them.

For everything else, Google’s security best practices centre and enterprise foundations blueprint cover it better than one article can.

Limits of the 4Cs

As promised, let’s dig into the limits of the 4Cs. On paper it feels MECE (mutually exclusive, collectively exhaustive). It isn’t collectively exhaustive. At least five gaps sit outside it:

  • What happens when a developer’s laptop is compromised?
  • Managed devices with EDR to catch credential theft, and short-lived credentials so there’s less worth stealing.
  • Identity is the new perimeter, so what stops a stolen login, or a compromised GitHub, from reaching the cluster?
  • SSO, phishing-resistant MFA and separate admin accounts.
  • Who protects the data once it’s copied somewhere the 4Cs doesn’t reach?
  • Classify it, and test your backups by restoring them.
  • Who owns each control when it quietly starts to rot?
  • Name an owner and an on-call for every one.
  • And what can an AI agent do with the permissions we’ve handed it, especially once someone slips a prompt injection into its context?
  • Give it its own least-privilege identity and log everything it does.
Press enter or click to view image in full size

Detection and response

The thing that stood out most in our scenario: where on earth was the response? Alerts fired across different tools, and nothing was contained until a human turned up on Monday.

So where should all that telemetry go, and how does it become action rather than a queue?

Enter the SIEM. On Google Cloud that’s Google Security Operations, which can detect, investigate and respond to threats.

Press enter or click to view image in full size

We won’t go too deep here (your tea has probably gone cold by now), but here’s how it would have handled Talha’s Saturday:

Press enter or click to view image in full size
  1. Normalise. Every log lands in one data model, so SecOps knows the pod and its two service accounts are the same workload.
  2. Detect. Now rules can spot sequences no single tool would see: curl hits the metadata server, then that pod’s identity lists buckets in a project it has never touched.
  3. Label. Each rule is tagged with a technique from MITRE ATT&CK, the shared catalogue of attacker moves. Credential Access, then Discovery, minutes apart: that progression is what makes it real.
  4. Respond. The labels feed a SOAR playbook, the automation that acts on alerts rather than filing them. Stolen credentials plus cross-project discovery on a production identity? Strip the IAM bindings, isolate the pod, page on-call.

The attacker gets the bucket list but probably not the copy, and someone finds out on Saturday instead of Monday.

MITRE has one more job: lay your rules over the ATT&CK matrix and the empty cells are moves an attacker can make without a single alert.

Top ten actions

Okay, so if you could only do ten things to secure your cluster, here are my recommendations:

Press enter or click to view image in full size

Tooling landscape

You can implement those 10 recommendations using Google-native capabilities, open source or commercial tooling, so there’s something for every budget.

Press enter or click to view image in full size

Back to Talha

Three months on, Talha ships another feature on a Wednesday afternoon. In the early hours of Thursday, an advisory lands for one of its dependencies, and once again the attackers read it first.

This time they land in a distroless container with no shell, running as a user who owns nothing. The pod’s identity can read one bucket. The runtime sensor kills the process and SecOps strips the service account’s roles, all before anyone’s alarm goes off.

Press enter or click to view image in full size

That’s what securing GKE end to end means. You won’t stop every bug at the Code layer. You make sure a bug in one layer stays a bug in one layer.

If you’ve got this far, thank you for sticking with it. I’d love to connect and talk with like-minded people, so if anything here resonated or you’d do it differently, reach out at hudeyfajama.com.

Sources

Tools that help: market recognition and recent changes

Kubernetes
Devsecops
Google Cloud Platform
Cybersecurity
Cloud Security

--

--

Hudeyfa Jama

Written by Hudeyfa Jama

37 followers
12 following

Engineer Substack - https://blog.thepattern.dev

Help

Status

About

Careers

Press

Blog

Store

Privacy

Rules

Terms

Text to speech