Let Copilot Troubleshoot Your EKS Cluster, but Not Delete It

Ajay Chinthapalli

7 min read

·

3 days ago

--

Listen

Share

Putting agentgateway between GitHub Copilot and the Amazon EKS MCP Server: create ✅ read ✅ delete ⛔

The Amazon EKS MCP Server is a great way to troubleshoot a cluster with an AI agent. Point GitHub Copilot at it, and the agent can list resources, read pod logs, pull CloudWatch metrics, check EKS Insights and search the EKS troubleshooting guide.

It can also delete things.

Start the server with --allow-write and the agent gets tools like manage_k8s_resource, which can create, update, patch and delete any Kubernetes resource. Add --allow-sensitive-data-access and it can read Secrets too. There’s no setting for “let it fix things, but never delete anything.”

So I put agentgateway in the middle. Copilot talks only to the gateway, the gateway launches the EKS MCP Server, and every tool call passes through two layers of policy on the way. The result: the agent can create and read, and a delete comes back as HTTP 403 before the EKS server ever sees it.

The full step-by-step guide, with every config file and command, is here:

https://gist.github.com/ajaychinthapalli/e7ccf99884a1b701b458caa7e07c31d3

Press enter or click to view image in full size

The setup

Everything runs on a laptop (I used macOS on Apple silicon) against an existing EKS cluster in us-east-2:

• VS Code with GitHub Copilot in Agent mode
• agentgateway 1.5.0, listening on http://localhost:3000/mcp
• Amazon EKS MCP Server (awslabs.eks-mcp-server), launched by the gateway over stdio with uvx
• Jaeger in Docker for traces (optional)

The gateway starts the EKS server itself, so there’s no separate server for Copilot to find:

cmd: uvx
args: ["--with", "botocore[crt]", "awslabs.eks-mcp-server@latest",
       "--allow-write", "--allow-sensitive-data-access"]

Both risky flags are on on purpose. The gateway is the control. (If you drop --allow-write, you get a read-only floor underneath the gateway’s policy, which is a good option too.)

Layer 1: which tools the agent can see

agentgateway’s mcpAuthorization policy decides which tools show up in tools/list. Copilot sees 13:

• 11 read-only diagnostic tools: list_k8s_resources, list_api_versions, get_k8s_events, get_pod_logs, get_cloudwatch_logs, get_cloudwatch_metrics, get_eks_metrics_guidance, get_eks_insights, get_eks_vpc_config, get_policies_for_role and search_eks_troubleshoot_guide
• 2 write tools, narrowed by layer 2: manage_k8s_resource and manage_eks_stacks

Three tools are hidden completely, and rejected if the agent calls them anyway:

• apply_yaml, because the gateway can’t inspect what’s inside the YAML file
• generate_app_manifest, because that’s authoring, not troubleshooting
• add_inline_policy, because IAM changes should be proposed by the agent and applied by a person

Layer 2: what the arguments are allowed to say

Hiding tools isn’t enough. manage_k8s_resource is one tool that does create, read, update, patch and delete. To allow some of that and not the rest, the gateway has to look at the arguments.

So the second layer is an authorization policy: a set of require expressions evaluated against the JSON-RPC request body. If any require fails, the gateway returns 403. It also fails closed: a malformed call or an evaluation error is denied, not waved through.

The five rules:

1. No JSON-RPC batches. A body starting with [ is refused, so several calls can’t be hidden inside one request.
2. No deletes. manage_k8s_resource must have operation != "delete".
3. No health-probe edits. A manage_k8s_resource call that mentions readinessProbe, livenessProbe or startupProbe is refused. Removing a probe is a quiet way to make a broken pod look healthy.
4. No Secrets. manage_k8s_resource and list_k8s_resources can’t target kind: Secret.
5. CloudFormation is read-only. manage_eks_stacks may only describe.

One version gotcha cost me time here. Older examples read mcp.tool.arguments in these rules. On agentgateway 1.5.x that isn’t available in route policies, so every deny check quietly returned 200. The rules now parse json(request.body) instead. The guide includes a one-line check: grep -c 'json(request.body)' config.yamlshould print 4.

The gateway only guards what goes through it

This was the most important lesson in the whole build.

Copilot’s Agent mode can run shell commands. If your terminal has a working kubeconfig, the agent can just run kubectl delete and skip the gateway entirely. No MCP call, no policy, no log line.

So I closed both bypasses:

1. Copilot can only reach the EKS server through the gateway. The project’s .vscode/mcp.json has a single entry pointing at http://localhost:3000/mcp. Any other MCP entry that launches awslabs.eks-mcp-server directly has to go.

2. VS Code’s terminal has no cluster access. A .vscode/settings.json in the project folder sets:

{
  "terminal.integrated.env.osx": { "KUBECONFIG": "/dev/null" },
  "chat.tools.autoApprove": false
}

Every terminal VS Code opens in that folder gets an empty kubeconfig, and VS Code asks before running any command. My normal macOS Terminal still works, so I can run kubectl, but the agent can’t. I also turn off the built-in terminal tools in Copilot’s tool picker.

If you skip this step, everything else in this post is decoration.

Testing it without touching a cluster

Before pointing anything at a real cluster, a smoke-test.sh script checks the guardrails against a cluster name that doesn’t exist (agw-smoke-test-does-not-exist). An allowed call gets “cluster not found” from the EKS server, and a denied call gets 403 from the gateway, so nothing real can change.

It runs 12 checks:

• 3 hidden-tool checks: apply_yaml, add_inline_policy and generate_app_manifest are not in tools/list
• 3 allow checks: read a DaemonSet, list pods in kube-system, patch a ServiceAccount’s IRSA annotation
• 6 deny checks: delete a pod, remove a readinessProbe, read a Secret, list Secrets, delete a CloudFormation stack, call the hidden apply_yaml

Expected result: 12 passed, 0 failed. Denied calls show http.status=403 in the gateway log, and the EKS server logs nothing for them, because it never received them.

The real test: create ✅ read ✅ delete ⛔

Then I asked Copilot to work against the real cluster, in a test namespace called agw-test:

• “Create an nginx deployment with 2 replicas.” Allowed: the gateway logs http.status=200 with eks.operation="create", and kubectl from my own terminal shows READY 2/2.
• “Show me the nginx deployment.” Allowed: 200 with eks.operation="read".
• “Delete the nginx deployment.” Blocked:

http.method=POST http.path=/mcp http.status=403 error="authorization failed" reason=AuthorizationThe deployment is still there. Listing Secrets is blocked the same way.

One surprise: when the gateway returns 403, VS Code treats it as a login challenge. It probes a few OAuth discovery URLs and then asks whether to register a client automatically. Click Cancel; the 403 is the gateway doing its job.

And if a delete ever does succeed, the gateway log tells you which bypass was used:

• No gateway log line at all: Copilot used kubectl in a terminal.
• eks.operation="delete" with a 200: the rules are the old mcp.tool.arguments kind.
• EKS tools in Copilot but no gateway line: there’s a direct server entry somewhere.

Seeing every call

Because everything goes through one place, observability comes almost for free:

• Audit log: one line per call in the gateway’s terminal, with the tool name, http.status and custom fields mapped from the arguments: eks.cluster, eks.operation, eks.kind, eks.namespace and eks.name.
• Traces: OTLP to Jaeger at localhost:4317, with the UI on http://localhost:16686.
• Metrics: Prometheus format on http://localhost:15020/metrics, including tool_calls_total.
• Admin UI: http://localhost:15000/ui, with the loaded config, the policies and an MCP playground.

Things that tripped me up

The guide has a 14-row troubleshooting table. The ones most people will hit:

• initialize returns 500 with a botocore[crt] error. If you sign in with aws login, the EKS server needs botocore[crt]. That’s why the launch command has --with botocore[crt].
• Copilot runs kubectl instead of the EKS tools. Usually the MCP server failed to start (look for a 500 on initialize), or terminal tools are still enabled.
• KUBECONFIG isn’t empty in VS Code. The settings file has to be in the folder you opened (not ~/.vscode), the folder has to be trusted, and you need a new terminal. echo $TERM_PROGRAM should print vscode.
• The policy suddenly changed. agentgateway watches its config file and reloads it. Saving a different config over it swaps the running policy immediately, so protect that file.
• A call you wanted got a 403. The probe and Secret rules match patterns in the request body, so they can catch an innocent request that merely mentions a probe. Read the request in the log and adjust the rule.
• The first connection times out. uvx downloads the server on first use. Run it once with --help beforehand.

What this is and isn’t

This is a local, single-user guardrail. It’s a fast way to let an AI agent help troubleshoot a cluster without handing it a loaded kubectl, and the whole thing runs on a laptop in under an hour.

It isn’t the last line of defence. The argument rules match the request body with expressions and patterns, and writes other than deletes (like patching a ServiceAccount annotation) are still allowed. For a shared or production cluster, I’d add Kubernetes RBAC on a dedicated identity, and admission policies that check what a write actually changes, so the cluster enforces the limits even if the gateway is bypassed.

But as a first step, three ideas carry most of the weight:

1. Hide the tools you can’t inspect.
2. Check the arguments of the ones you keep, and fail closed.
3. Close every path that doesn’t go through the gateway.

The complete guide, with the full gateway config, smoke test, VS Code settings, cleanup steps and troubleshooting table:

https://gist.github.com/ajaychinthapalli/e7ccf99884a1b701b458caa7e07c31d3

How are you keeping AI agents in check around your clusters? I’d love to hear what’s worked for you in the comments.