Building an AI Agent Mesh on Amazon EKS: Discover, Call, Govern

Ajay Chinthapalli

6 min read

·

3 days ago

--

Listen

Share

An on-call agent that finds the right specialist in agentregistry and delegates to it over A2A, with every call going through agentgateway and every model call going to Amazon Bedrock without a single stored key.

Most AI agent demos have one agent. Real operations won’t.

You’ll want an agent that knows Kubernetes troubleshooting, another that knows cost, another for cluster upgrades, another for CI failures. And then you hit the questions nobody’s demo answers:

• How does one agent find another? Hard-coding agent names and URLs works for two agents. It doesn’t work for twenty.
• Who is allowed to call whom? An agent that can call any other agent is a new lateral-movement path.
• Where do the model credentials live? Every agent with its own API key is every agent with a secret to leak.

So I built a small agent mesh on Amazon EKS to answer them. A Platform On-call Agent receives an alarm, searches agentregistry for the right specialist, and calls it over A2A (the agent-to-agent protocol) through agentgateway. Every model call goes through the same gateway to Amazon Bedrock, using EKS Pod Identity instead of stored keys.

It ran end to end on an existing EKS 1.36 cluster. The code, scripts and terminal logs, including the failures, are on GitHub:

👉 https://github.com/ajaychinthapalli/eks-agent-mesh-agentregistry

Press enter or click to view image in full size

The pattern: discover, get the card, communicate

The idea is simple, and it’s the same one service meshes and service catalogs solved for microservices:

1. Discover. The On-call Agent asks agentregistry for an agent that matches the task.
2. Get the Agent Card. The registry returns the matching agent and its card.
3. Communicate. The On-call Agent calls that agent over A2A, through the gateway.

The pilot runs one specialist, an EKS Troubleshooter. The diagram also shows where a Cost Optimizer, an Upgrade Advisor and a GitHub Actions agent would plug in. That’s the point of a registry: publish a new specialist, and the On-call Agent can find it with no code change.

Tech stack

• Amazon EKS 1.36 in us-east-2, with EKS Pod Identity
• agentregistry v0.4.0, a catalog of agents, MCP servers and skills, with discovery over MCP and its catalog on Amazon RDS for PostgreSQL
• agentgateway v1.6.0, a Gateway API proxy for A2A, MCP and LLM traffic, on an internal NLB
• kagent 0.10.3 to run both agents as Kubernetes resources and serve A2A
• Amazon Bedrock with Claude Sonnet 4.5 for both agents
• Amazon ECR for the On-call Agent’s image

The two agents are built differently on purpose. The EKS Troubleshooter is a declarative kagent agent with read-only kubectl tools. The On-call Agent is a bring-your-own agent written in Python with Google’s ADK, scaffolded by arctl(agentregistry’s CLI) and deployed through the registry.

Press enter or click to view image in full size

How one incident flows

1. The alarm reaches agentgateway with the caller’s Kubernetes ServiceAccount token.
2. The gateway checks the token and routes the A2A call, by path, to the On-call Agent in kagent.
3. The On-call Agent searches agentregistry over MCP and finds ekstroubleshooter.
4. It calls the Troubleshooter back through the gateway, using its own ServiceAccount token.
5. The Troubleshooter reads pods, events and logs with read-only kubectl tools and returns a root cause and a fix.

The On-call Agent’s instructions are short and strict: search the registry for an agent whose description matches the task, don’t guess agent names, call the best match, and report which agent was used, the root cause, the evidence and the fix. It never changes the cluster itself. If no registered agent fits, it says so.

Governance lives in the gateway

Every hop goes through agentgateway, so that’s where the rules are:

• Identity: callers authenticate with their Kubernetes ServiceAccount tokens. The gateway verifies them as JWTs against the cluster’s EKS OIDC issuer. No shared secrets between agents.
• Allow-lists per route: a CEL rule on each A2A route says who may call it. Only the On-call Agent’s ServiceAccount may call the Troubleshooter:

jwt.sub == 'system:serviceaccount:kagent:platformoncall-latest-platformoncall'

• Model access: both agents call Bedrock through the gateway’s OpenAI-compatible route. The gateway holds the AWS identity through Pod Identity, so neither agent has AWS credentials. In the On-call Agent’s code, the API key is literally the string "unused".

The result

I sent a real alarm: “checkout pods in namespace shop are in CrashLoopBackOff. Find the root cause and propose a fix.”

The On-call Agent queried the registry over MCP, picked ekstroubleshooter, and delegated through the gateway. The answer came back with:

• Agent used: ekstroubleshooter
• Root cause: the checkout pods were OOMKilled. The container tries to allocate 250M of memory, and its limit is 100Mi.
• Evidence: OOMKilled with exit code 137, CrashLoopBackOff with a dozen-plus restarts, and the mismatch between the allocation and the limit
• Fix: raise the memory limit, or reduce the allocation if resources are tight

The gateway log showed every hop in order: several route=bedrock … 200 model calls, then route=a2a-eks-troubleshooter … completed, then route=a2a-platform-oncall … completed.

And the security checks passed:

• No token to the Troubleshooter: 401
• Valid token, wrong caller: 403
• No token to the On-call Agent: 401
• Authorized caller, end to end: the OOMKilled diagnosis

What bit me

The repo has a full list of field-tested corrections. The ones you’re most likely to hit:

Solo.io’s EKS add-ons weren’t published for Kubernetes 1.36 yet. I installed the upstream open-source Helm charts instead, and deleted kagent’s demo agents.

Bedrock was called with the node’s IAM role. The first gateway proxy pod had started before its Pod Identity association existed. A restart fixed it, and I confirmed AWS_CONTAINER_CREDENTIALS_FULL_URI in the new pod. (This is the second project in a row where this one got me. Always restart after creating the association.)

Bedrock rejected a tool with an empty description. kagent’s ADK base image strips Python docstrings, and Bedrock requires every tool to have a description. The fix is one line: set call_agent.__doc__ explicitly.

The CloudWatch add-on crashed my agent. The Observability add-on’s Application Signals auto-instrumentation injected a Python agent into the new pod, which failed with ImportError: cannot import name 'LogData'. I excluded the kagent, agentgateway-system and agentregistry namespaces from it.

arctl rejects hyphens in agent names. Registry names and gateway paths are ekstroubleshooter and platformoncall. Kubernetes names keep their hyphens.

The registry doesn’t discover agents you create with kubectl. I registered a record for the Troubleshooter with a keyword-rich title and description. Registry search matches that text, so the description is effectively the agent’s API: write it for the agent that will search for it.

Don’t trust the card URL for routing. The A2A card’s url pointed at the kagent controller, not the gateway. The On-call Agent builds the gateway path from the registry name instead, so every call still goes through the gateway’s checks.

Before production

This is a pilot, and I took shortcuts that I’d fix first:

• Add TLS on the Gateway listener (it’s HTTP inside the VPC).
• Use projected ServiceAccount tokens with an agentgateway audience; the JWT policy currently accepts any audience.
• Scope the Bedrock IAM policy to specific model ARNs.
• Move kagent off its bundled PostgreSQL to an external database.
• Use private subnets with NAT; this VPC only had public subnets.

Try it

You need an EKS cluster with the Pod Identity Agent and EBS CSI add-ons, room for about 20 extra pods, subnets tagged for internal load balancers, and Bedrock model access. Then:

cp scripts/00-env.example.sh scripts/00-env.sh   # fill in your values
source scripts/00-env.sh
./scripts/01-platform.sh          # LB controller, agentgateway, kagent
./scripts/02-gateway-bedrock.sh   # internal NLB Gateway, Pod Identity, Bedrock route
./scripts/03-agentregistry.sh     # RDS + agentregistry
./scripts/04-troubleshooter.sh    # demo workload, Troubleshooter, route, registry record
./scripts/05-oncall.sh            # build, publish and deploy the On-call Agent
./scripts/06-security.sh          # JWT + caller allow-lists
./scripts/07-test.sh              # 401/403 checks and the end-to-end incident

99-cleanup.sh removes everything the pilot added.

👉 https://github.com/ajaychinthapalli/eks-agent-mesh-agentregistry

The takeaway

Agents need the same things microservices needed: a catalog to find each other, an identity to prove who’s calling, and a gateway that enforces who may talk to whom. agentregistry, A2A and agentgateway give you all three, and on EKS you can build them on identities you already have, ServiceAccount tokens and Pod Identity, with no API keys to manage.

Next, I want to add the Cost Optimizer and Upgrade Advisor specialists and see whether the On-call Agent really can pick them up with no code change. If you’re building multi-agent systems, how are you handling discovery and trust between agents? Let me know in the comments.