How to build AI agents that survive failures, resume interrupted tasks, handle approvals, and execute reliably in production.

1. Why Do AI Agents Need Durable Execution?

Most AI applications start with a relatively straightforward execution model.

A user sends a message, the application invokes an LLM, the model may call a tool, and the application returns a response.

For simple interactions, this is usually sufficient.

However, agentic applications introduce a different level of complexity.

An agent may execute several model calls, invoke multiple tools, delegate tasks to other agents, wait for external systems, and require human approval before continuing.

Consider an agent responsible for reviewing and processing a business request:

1. The user submits a request.
2. The agent analyzes the request.
3. It retrieves information from internal systems.
4. It delegates an analysis task to a specialist agent.
5. It generates a proposed action.
6. It requests approval from an authorized user.
7. After approval, it executes the operation.
8. It records the result and completes the task.

This workflow may take seconds, minutes, or even several hours.

Now imagine the application crashes after step five.

What happens to the work already completed?

Does the agent start over? Are previous model calls executed again? Will the user need to submit the request again? Could an external operation be executed twice?

These are not unusual edge cases. They are fundamental engineering problems in long-running AI systems.

A conventional in-memory agent generally keeps its execution state inside the current process. When that process terminates unexpectedly, the state may be lost.

Saving conversation messages to a database does not solve this problem by itself.

Durable Execution is the ability to preserve workflow progress and resume execution from a known point after failures, interruptions, or external events.

It allows us to build agents that are not dependent on one uninterrupted application process.

The objective is not merely to make an agent remember what happened. It is to make the entire execution reliable.

2. State Persistence vs. Durable Execution

These concepts are related, but they are not interchangeable.

State Persistence

State persistence means storing information so that it can be retrieved later.

For an AI application, this might include:

• Conversation history
• User preferences
• Current task status
• Agent configuration
• Intermediate results
• Retrieved documents
• Tool execution results

For example, we can store conversation messages in PostgreSQL and load them when a user returns to the application.

This provides conversation continuity.

However, storing messages does not automatically tell us where an interrupted workflow should resume.

Durable Execution

Durable execution goes further.

It records the execution progress and the results of completed steps so that the workflow can continue without repeating all previous work.

Consider this example:

Agent Workflow
      |
      v
Retrieve Context
      |
      v
LLM Analysis
      |
      v
Execute Tool
      |
      v
Generate Report
      |
      X Application Crashes

Without durable execution, the application may need to restart the entire workflow.

With checkpointing:

Agent Workflow
      |
      v
Retrieve Context       ✓ Checkpoint
      |
      v
LLM Analysis           ✓ Checkpoint
      |
      v
Execute Tool           ✓ Checkpoint
      |
      v
Generate Report        X Failed
      |
      v
Application Restarts
      |
      v
Resume from last
completed checkpoint
      |
      v
Generate Report
      |
      v
Complete

The completed results can be reused while the failed operation is retried.

This distinction becomes particularly important when model calls are expensive or when tools perform operations against external systems.

What about Agent Memory?

Memory and durable execution are also different concepts.

ConceptResponsibilityConversation historyWhat the user and agent discussedShort-term memoryInformation needed during the current interactionLong-term memoryInformation retained across interactionsCheckpointingWhere execution stopped and what has completedDurable executionHow the workflow resumes and continues reliably

For example, an agent may remember a user’s preferences through long-term memory while a durable workflow maintains the execution state of an ongoing approval process.

One does not replace the other.

Reference: LangGraph — Persistence

3. Core Concepts Behind Durable Execution

Before choosing a framework, it helps to understand the underlying engineering primitives.

3.1. Checkpointing

A checkpoint is a persisted representation of workflow state at a specific point in execution.

Depending on the runtime, it may include:

• Completed step identifiers
• Step outputs
• Workflow state
• Execution metadata
• Pending operations
• Current task status
• References to external payloads

When a workflow resumes, the runtime uses these checkpoints to determine which operations have already completed.

Checkpointing is particularly useful when a workflow contains expensive or long-running operations.

However, we should avoid storing unnecessarily large objects in every checkpoint.

For example, instead of persisting a 50 MB document inside workflow state, we can store it in object storage and keep only its identifier and metadata in the checkpoint.

3.2. Replay

Replay is the process of reconstructing workflow execution using its previously recorded history.

Some durable execution engines replay workflow code to rebuild the current execution state.

This introduces an important engineering requirement: deterministic execution.

If the same workflow is replayed with the same history, it should make the same workflow decisions.

Consider:

async function workflow() {
  const timestamp = Date.now();

  // Continue execution...
}

If the workflow is replayed later, Date.now() returns a different value.

This could change its execution path.

Likewise, direct network calls, random values, and database queries should not be performed inside deterministic workflow definitions when the runtime expects replay-safe code.

These operations should be isolated into activities or durable steps.

For example:

async function workflow() {
  const timestamp = await getCurrentTimeStep();

  // Continue using the recorded result...
}

The durable runtime records the step result and reuses it during replay.

This is a foundational concept in systems such as Temporal.

3.3. Durable Steps

A durable workflow is usually divided into independent execution units.

For example:

Workflow
   |
   +-- Step 1: Retrieve documents
   |
   +-- Step 2: Generate analysis
   |
   +-- Step 3: Validate output
   |
   +-- Step 4: Request approval
   |
   +-- Step 5: Execute operation
   |
   +-- Step 6: Save final result

Each completed step produces a checkpoint.

If step five fails, the workflow should not need to repeat document retrieval, model analysis, and validation.

This also allows us to apply different retry policies to different operations.

3.4. Durable Waits

A workflow may need to wait for something that is outside the application’s control.

Examples include:

• Human approval
• An external webhook
• A scheduled execution time
• A payment confirmation
• A response from another agent
• Completion of a long-running background operation

A regular asynchronous function can wait for an event, but keeping the process alive indefinitely is not a scalable approach.

Durable execution systems can suspend the workflow, persist its state, and resume it when the expected event arrives.

The application does not need to keep a worker occupied during the waiting period.

3.5. Retries and Recovery

Failures should be handled according to their type.

A network timeout, provider rate limit, validation error, and authorization failure should not necessarily follow the same retry policy.

A retry policy should define:

• Maximum attempts
• Initial delay
• Backoff strategy
• Maximum delay
• Retryable error categories
• Overall execution deadline
• Cancellation behavior

A durable execution engine provides the infrastructure for retries, but the application still needs to define what should be retried and why.

4. The Most Important Concept: Idempotency

This is one of the most important aspects of building reliable AI agents.

Durable execution can resume workflows and retry failed steps. But retries can introduce duplicate side effects.

Consider an agent that generates a report and publishes it to an external system.

The operation succeeds, but the application crashes before the workflow records the result.

When execution resumes, the workflow may attempt to publish the report again.

Without additional protection, we could create two reports.

This is why idempotency matters.

An operation is idempotent when executing it multiple times with the same logical input produces the same intended outcome as executing it once.

For example, consider a report publishing API:

async function publishReport(input: {
  reportId: string;
  content: string;
  operationId: string;
}) {
  const response = await fetch(
    'https://api.example.com/reports',
    {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
        'Idempotency-Key': input.operationId,
      },
      body: JSON.stringify({
        reportId: input.reportId,
        content: input.content,
      }),
    },
  );

  if (!response.ok) {
    throw new Error(
      `Report publishing failed: ${response.status}`,
    );
  }
  return response.json();
}

The important detail is operationId.

It must be generated and persisted by a trusted part of the application before the external operation is attempted. It should not be an arbitrary identifier generated by the LLM.

The receiving service must also implement idempotency-key handling. Merely sending the header does not guarantee duplicate prevention.

Why isn’t exactly-once execution automatic?

This is a common misunderstanding.

A workflow engine can guarantee that a successfully checkpointed step is not unnecessarily executed again during replay.

However, an external side effect and the workflow’s checkpoint cannot always participate in the same atomic transaction.

There is a failure window:

1. An external API accepts the request.
2. The external operation succeeds.
3. The application crashes.
4. The workflow has not recorded the successful result.
5. The workflow retries.

This means we should not assume that every external operation executes exactly once.

A more accurate approach is to design for:

• At-least-once execution attempts.
• Idempotent external operations.
• Durable operation identifiers.
• Deduplication at the receiving service.
• Transactional outbox patterns where appropriate.
• Reconciliation for operations whose result is uncertain.

For critical business operations, this distinction is non-negotiable.

A durable workflow is not a substitute for transactional business design.

5. Human-in-the-Loop: When an Agent Needs Approval

Not every operation should be executed automatically.

In enterprise applications, an agent may need authorization before performing an operation that has financial, operational, or security implications.

For example:

• Publishing a report
• Updating customer information
• Executing a database modification
• Approving a business transaction
• Changing production configuration
• Triggering an external workflow

A traditional implementation may look like this:

Agent
  |
  v
Generate Proposed Action
  |
  v
Save Pending Approval
  |
  v
Notify User
  |
  v
Wait for Response
  |
  v
User Approves
  |
  v
Continue Agent

The problem is that the application needs to preserve enough state to resume the original execution.

A durable workflow makes this much easier.

It can:

1. Execute the agent until an approval boundary.
2. Persist the current state.
3. Suspend execution.
4. Notify the appropriate user.
5. Wait for an approval event.
6. Resume from the saved state.
7. Continue or abort based on the decision.

The approval process may take seconds, hours, or days.

Importantly, the application should verify the identity and authorization of the person responding. The decision should be bound to the original operation, its parameters, and its expiration policy.

An approval should not be a generic boolean that can be reused for another action.

6. Implementing Durable Agents with Vercel AI SDK

For TypeScript-based applications, the Vercel AI SDK ecosystem provides an interesting integration through WorkflowAgent.

The standard ToolLoopAgent runs its execution loop in memory.

This is suitable for short interactions, but its progress can be lost when the underlying process terminates.

WorkflowAgent, provided through @ai-sdk/workflow, runs the agent inside the Vercel Workflow runtime.

It provides:

• Persistent execution progress
• Independent tool retries
• Recovery after process interruptions
• Human approval that can survive long waits
• Workflow-level execution visibility
• Resumable chat streaming through WorkflowChatTransport

An important distinction: durability comes from the Workflow runtime, not simply from changing the agent class.

Step 1: Install the packages

pnpm add ai @ai-sdk/workflow workflow zod

Step 2: Define durable tool operations

A tool operation can be marked as a durable step using the 'use step' directive.

async function searchDocumentsStep(input: {
  query: string;
}) {
  'use step';
  const response = await fetch(
    `https://api.example.com/search?q=${encodeURIComponent(
      input.query,
    )}`,
  );
  if (!response.ok) {
    throw new Error(
      `Document search failed: ${response.status}`,
    );
  }
  return response.json();
}

The runtime can persist the result of this step and retry it independently when a retryable failure occurs.

This is particularly useful when the operation interacts with an unreliable external service.

Step 3: Create a WorkflowAgent

import {
  WorkflowAgent,
  type ModelCallStreamPart,
} from '@ai-sdk/workflow';
import {
  convertToModelMessages,
  tool,
  type UIMessage,
} from 'ai';
import { getWritable } from 'workflow';
import { z } from 'zod';

async function searchDocumentsStep(input: {
  query: string;
}) {
  'use step';
  const response = await fetch(
    `https://api.example.com/search?q=${encodeURIComponent(
      input.query,
    )}`,
  );

  if (!response.ok) {
    throw new Error('Document search failed');
  }

  return response.json();
}

async function publishReportStep(input: {
  reportId: string;
}) {
  'use step';
  const response = await fetch(
    'https://api.example.com/reports/publish',
    {
      method: 'POST',
      headers: {
        'Content-Type': 'application/json',
      },
      body: JSON.stringify(input),
    },
  );

  if (!response.ok) {
    throw new Error('Report publishing failed');
  }

  return response.json();
}

export async function runAssistant(
  messages: UIMessage[],
) {
  'use workflow';

  const agent = new WorkflowAgent({
    model: 'anthropic/claude-sonnet-4-6',
    instructions: `
      You are an enterprise research assistant.
      Search internal documentation when needed.
      Prepare reports based on verified information.
      Never publish a report without approval.
    `,
    tools: {
      searchDocuments: tool({
        description:
          'Search internal documentation.',
        inputSchema: z.object({
          query: z.string(),
        }),
        execute: searchDocumentsStep,
      }),
      publishReport: tool({
        description:
          'Publish a prepared report.',
        inputSchema: z.object({
          reportId: z.string(),
        }),
        needsApproval: true,
        execute: publishReportStep,
      }),
    },
  });

  const result = await agent.stream({
    messages: convertToModelMessages(messages),
    writable: getWritable<ModelCallStreamPart>(),
  });

  return {
    messages: result.messages,
  };
}

This example demonstrates the core pattern:

• The workflow function is marked with 'use workflow'.
• Tool implementations that need durable retries are marked with 'use step'.
• The report publishing tool requires human approval.
• The workflow can suspend and resume while waiting for the approval decision.

For a real deployment, the publishing operation should also implement server-controlled idempotency and authorization. The simple example focuses on the workflow integration rather than the complete business security layer.

Step 4: Understand the difference between WorkflowAgent and ToolLoopAgent

FeatureToolLoopAgentWorkflowAgentExecutionIn-memoryDurable workflowProcess restart recoveryNo automatic recoverySupported by runtimeTool retriesApplication-managedPer-step retry supportLong approval waitsRequires custom handlingDurable suspensionTool execution visibilityAI SDK telemetryWorkflow step visibilityStreamingDirect stream consumptionWorkflow writable

The recommended approach is not to replace every ToolLoopAgent with WorkflowAgent.

For short, synchronous agent interactions, an in-memory agent is often simpler.

Use durable execution when the workflow needs to outlive its original request, survive process interruptions, wait for external events, or retry individual operations independently.

Step 5: Make Chat Streaming Resumable

One challenge in long-running AI applications is that the backend workflow and the frontend connection have different lifecycles.

A user might:

• Refresh the browser.
• Lose network connectivity.
• Navigate away from a page.
• Reconnect after a function timeout.

A durable workflow may still be running even though the original HTTP connection has disappeared.

WorkflowChatTransport provides a mechanism for resuming the chat stream using the workflow run ID and previously received stream chunks.

A simplified client configuration:

'use client';

import { useChat } from '@ai-sdk/react';
import { WorkflowChatTransport } from '@ai-sdk/workflow';
import { useMemo } from 'react';

export default function Chat() {
  const transport = useMemo(
    () =>
      new WorkflowChatTransport({
        api: '/api/chat',
        maxConsecutiveErrors: 5,
        initialStartIndex: -50,
      }),
    [],
  );

  const { messages, sendMessage } = useChat({
    transport,
  });
  // Render conversation and handle user input.
}

The server also needs the corresponding workflow stream endpoints.

This is an important architectural detail: a durable backend does not automatically make the browser connection durable. The frontend needs a way to reconnect and retrieve the appropriate part of the workflow stream.

References: Vercel — WorkflowAgent · AI SDK — WorkflowAgent Migration

7. LangGraph: Checkpointing and Interrupts

LangGraph approaches durable agent execution through graph state and persistence.

Its persistence architecture provides two complementary mechanisms:

Checkpointers

Persist graph state for a specific thread.

They support:

• Conversation continuity
• Recovery from interruptions
• Human-in-the-loop workflows
• Time travel
• Fault tolerance

Stores

Persist application-defined information across different threads.

They are useful for:

• User preferences
• Long-term memory
• Shared knowledge
• Cross-conversation information

A simplified configuration:

import {
  MemorySaver,
  StateGraph,
  Annotation,
} from '@langchain/langgraph';

const State = Annotation.Root({
  messages: Annotation<string[]>({
    reducer: (current, update) => [
      ...current,
      ...update,
    ],
    default: () => [],
  }),
});

const builder = new StateGraph(State);
// Add nodes and edges...
const checkpointer = new MemorySaver();

const graph = builder.compile({
  checkpointer,
});

const result = await graph.invoke(
  {
    messages: ['Analyze the requested document.'],
  },
  {
    configurable: {
      thread_id: 'analysis-123',
    },
  },
);

This example uses an in-memory checkpointer for illustration.

It is not suitable for surviving process restarts. For production, use a persistent implementation such as PostgreSQL or another supported durable checkpointer.

Human-in-the-loop with interrupts

LangGraph provides the interrupt() function to pause a graph and wait for external input.

Conceptually:

import {
  interrupt,
  Command,
} from '@langchain/langgraph';

async function approvalNode(state: {
  proposedAction: string;
}) {
  const decision = interrupt({
    action: state.proposedAction,
    message: 'Approval is required.',
  });

  if (!decision.approved) {
    return {
      status: 'rejected',
    };
  }

  return {
    status: 'approved',
  };
}

The graph persists its state through its checkpointer.

Later, the application can resume the same thread using Command with the user's response.

This is useful for workflows where execution must pause for approval or additional input.

One subtle but important implementation detail: when resuming an interrupted graph, the interrupted node may execute again from its beginning. Operations performed before interrupt() therefore need to be designed carefully to avoid duplicate side effects.

References: LangGraph Persistence · LangGraph Interrupts

8. Temporal: Durable Execution as an Infrastructure Primitive

Temporal takes a different approach.

Rather than focusing specifically on agent state or conversation management, Temporal provides a general-purpose durable execution platform.

It is designed for long-running, stateful workflows that need to survive failures.

Two concepts are particularly important:

Workflow

Defines the orchestration logic and execution history.

Activity

Performs external or non-deterministic operations such as API requests, database access, LLM calls, and file operations.

A simplified AI workflow could look like this:

Temporal Workflow
       |
       v
Retrieve Context Activity
       |
       v
LLM Analysis Activity
       |
       v
Validation Activity
       |
       v
Wait for Approval Signal
       |
       v
Execute Business Operation
       |
       v
Complete

Temporal records workflow history and uses deterministic replay to reconstruct execution state.

External interactions belong in Activities so their results can be recorded and reused during replay.

This separation is fundamental.

An LLM invocation should not be performed directly in a deterministic workflow definition. It should be executed as an Activity.

The same applies to database calls and external HTTP requests.

When does Temporal make sense?

Temporal is particularly relevant when:

• Workflows span multiple services.
• Tasks may run for hours or days.
• Strong recovery requirements exist.
• Human approval is part of the process.
• Many independent workers execute tasks.
• Workflow execution is critical to business operations.

It also introduces additional infrastructure and operational responsibilities.

Teams need to understand workflow history, Activity retries, Worker deployment, workflow versioning, and task queue management.

References: Temporal — Durable AI · Temporal TypeScript SDK · Temporal Workflow Determinism

9. Inngest and DBOS: Other Approaches Worth Evaluating

Temporal and LangGraph are not the only options.

For TypeScript-based applications, two other projects are worth considering.

Inngest

Inngest provides durable functions and agent workflows.

Its execution model is built around steps.

For example:

import { inngest } from './client';

export const researchWorkflow = inngest.createFunction(
  {
    id: 'research-workflow',
  },
  async ({ event, step }) => {
    const context = await step.run(
      'retrieve-context',
      async () => {
        return retrieveContext(event.data.query);
      },
    );

    const analysis = await step.run(
      'generate-analysis',
      async () => {
        return generateAnalysis(context);
      },
    );

    const report = await step.run(
      'validate-report',
      async () => {
        return validateReport(analysis);
      },
    );

    return report;
  },
);

The runtime checkpoints completed steps and resumes from the failed step when execution recovers.

Inngest also provides step.waitForEvent() for human approvals and other external events.

Its durable agent approach goes further by treating each model call and tool call as a separately checkpointed step.

This is useful when we want to write ordinary TypeScript functions instead of defining an explicit graph.

DBOS

DBOS provides durable execution for TypeScript and other languages, using a database-backed workflow model.

Its TypeScript SDK offers:

• Workflow registration
• Durable steps
• Configurable retries
• Workflow IDs
• Idempotency and deduplication mechanisms
• Background execution
• Recovery after process restarts

A simplified example:

import { DBOS } from '@dbos-inc/dbos-sdk';

async function retrieveDocuments() {
  // External API or database operation
  return ['document-1', 'document-2'];
}

async function analyzeDocuments() {
  // LLM interaction
  return {
    summary: 'Analysis completed',
  };
}

async function researchWorkflow() {
  const documents = await DBOS.runStep(
    () => retrieveDocuments(),
    {
      name: 'retrieve-documents',
      retriesAllowed: true,
      maxAttempts: 3,
    },
  );

  const analysis = await DBOS.runStep(
    () => analyzeDocuments(),
    {
      name: 'analyze-documents',
      retriesAllowed: true,
      maxAttempts: 3,
    },
  );

  return {
    documents,
    analysis,
  };
}

const workflow = DBOS.registerWorkflow(
  researchWorkflow,
);

This example shows the basic step model. A production application must initialize DBOS, register the workflow, and configure its database and worker lifecycle.

DBOS also requires deterministic workflow code and appropriate serialization of workflow inputs and outputs.

How do these approaches compare?

TechnologyMain strengthBest suited forLangGraphGraph state and checkpointingStateful agent workflowsTemporalGeneral-purpose durable executionComplex distributed business processesInngestDurable TypeScript functions and agentsEvent-driven agent workflowsDBOSDatabase-backed durable executionTypeScript applications needing durable stepsVercel Workflow + WorkflowAgentNative AI SDK integrationDurable agents in Vercel-based applications

There is no universally correct choice.

The right decision depends on the execution model, deployment environment, workflow duration, infrastructure constraints, and the level of operational control required.

References: Inngest Durable Agents · DBOS TypeScript Workflows

10. Designing a Production-Ready Agent Workflow

Choosing a durable execution framework is only one part of the solution.

We also need to design the workflow itself.

Let’s consider a document analysis and approval workflow.

The system should:

1. Receive a document.
2. Extract its content.
3. Analyze the information.
4. Generate a proposed report.
5. Wait for human approval.
6. Publish the approved report.
7. Store the final result.

A possible architecture:

             User
              |
              v
        AI Application
              |
              v
       Workflow Runtime
              |
      +-------+--------+
      |                |
      v                v
  Extract Step    Analysis Step
      |                |
      v                v
   Storage          LLM / Tools
      |                |
      +--------+-------+
               |
               v
        Validation Step
               |
               v
        Approval Required
               |
        +------+------+
        |             |
      Reject        Approve
        |             |
        v             v
      Stop      Publish Step
                      |
                      v
                 Save Result

Several design decisions matter here.

Persist the right information

Do not put every document, model response, or binary object into the workflow’s event history.

Use references to external storage for large payloads.

Persist only the data required to recover the next step.

Separate planning from execution

The LLM may decide which operation should happen next, but the runtime should control which operations are actually executed.

The agent should not own unrestricted workflow persistence or authorization decisions.

Define explicit failure boundaries

A document extraction failure should not necessarily require the analysis to restart.

An approval timeout should not cause the document to be processed again.

A publishing failure should not require repeating the original LLM request.

Each operation should have a clear recovery strategy.

Keep business state authoritative

A workflow checkpoint records execution progress.

It should not replace the authoritative transaction state of the business system.

For example, an approval workflow may persist its state in the orchestration engine, but the actual transaction should be committed and recorded in the relevant business system.

Make workflows version-aware

Long-running workflows can remain active across application deployments.

A workflow that started using one version of the code may resume after a newer version has been deployed.

Changing step ordering, workflow logic, or serialized state formats can break recovery if compatibility is not considered.

Workflow versioning, migration strategies, and backward-compatible payloads should be part of the deployment design.

11. Common Mistakes in Durable Agent Implementations

There are a few mistakes I would specifically avoid.

1. Treating conversation history as workflow state

Saving messages does not mean the agent can resume from the exact point where it stopped.

2. Retrying everything

A failed tool should not force all previous model calls and tool executions to run again.

Retry boundaries should be explicit.

3. Assuming exactly-once side effects

A durable workflow cannot automatically guarantee exactly-once behavior across arbitrary external services.

Use idempotency keys, deduplication, and reconciliation.

4. Storing large payloads inside checkpoints

Store references to documents and large artifacts instead of repeatedly serializing them.

5. Ignoring workflow versioning

A long-running workflow may outlive multiple application deployments.

Plan how in-flight executions will behave when code changes.

6. Keeping workers alive while waiting for approval

A durable wait should suspend execution and release the worker rather than holding a process open.

7. Using durable execution for every interaction

A simple question-answering endpoint does not necessarily need a workflow engine.

The added infrastructure should solve an actual reliability problem.

8. Confusing agent reasoning with business authorization

An agent deciding to perform an action does not mean that action is authorized.

Sensitive operations must be validated and approved by trusted application logic.

12. What Can an AI Chapter Lead Contribute?

I believe durable execution is particularly important when multiple teams are building AI-powered features within the same organization.

Without shared patterns, each team may implement its own persistence, retry logic, background execution, and approval mechanisms.

This creates duplicated engineering effort and inconsistent reliability.

There are several practical areas where an AI Chapter Lead can contribute.

1. Establish a Common Durable Execution Pattern

Define when an agent should run in memory and when it should run inside a durable workflow.

For example:

Use caseRecommended approachSimple chat responseIn-memory agentShort tool interactionStandard agent with controlled retriesMulti-step background taskDurable workflowHuman approvalDurable wait and resumeLong-running researchDurable agentCross-service business processWorkflow engineIndependent subagent tasksDurable child workflows or delegated executions

This prevents unnecessary infrastructure while ensuring that critical workflows have appropriate recovery mechanisms.

2. Build a Shared Workflow Abstraction

We can create a reusable internal package that provides:

• Workflow lifecycle management
• Checkpoint persistence
• Standard retry policies
• Idempotency helpers
• Cancellation support
• Human approval mechanisms
• Execution state management
• Observability integration

Teams should not need to implement these capabilities independently.

However, shared abstractions should not hide framework-specific behavior that affects reliability.

3. Define Standard Retry and Recovery Policies

Not every failure should trigger another attempt.

For example:

FailureStrategyTemporary network failureRetry with backoffProvider rate limitRespect retry guidance and budgetInvalid model outputValidate and optionally regenerateAuthorization failureStop and escalateInvalid tool argumentsCorrect or request clarificationBusiness rule violationStop or request human reviewExternal operation timeoutReconcile before retrying

We can define common retry budgets, deadlines, maximum attempts, and error classification rules.

This reduces inconsistent failure handling across applications.

4. Establish Idempotency Standards

Every operation that creates an external side effect should have an explicit idempotency strategy.

For example:

• Database writes
• Report publishing
• Email sending
• External API requests
• Business transaction execution
• Agent-to-agent task submission

This is especially important when workflows involve retries or uncertain execution outcomes.

5. Standardize Human Approval

We can establish a shared approval model that defines:

• Who can approve an action.
• Which operations require approval.
• How approval requests are stored.
• How decisions are associated with workflow executions.
• What happens when approval expires.
• How rejected actions are handled.
• How approval decisions are audited.

This makes human-in-the-loop workflows consistent across teams.

6. Build a Reference Implementation

A practical internal example should demonstrate:

1. Durable agent execution.
2. Step-level checkpointing.
3. Independent retries.
4. Human approval.
5. Resume after application restart.
6. Idempotent external operations.
7. Distributed tracing.
8. Workflow versioning.

This is more useful than a standalone demo that only shows a successful agent response.

7. Measure Reliability

We should define measurable outcomes.

Useful metrics include:

• Workflow completion rate
• Recovery success rate
• Number of repeated model calls
• Duplicate side-effect attempts
• Approval wait duration
• Retry count per workflow
• Average workflow duration
• Cost per successful workflow
• Human escalation rate

The objective should be to understand whether durable execution actually improves reliability without creating unnecessary complexity.

13. When Should We Adopt Durable Execution?

A durable execution framework is particularly valuable when:

• A task involves multiple dependent operations.
• Execution can exceed HTTP request timeouts.
• A workflow must survive process restarts.
• Expensive model calls should not be repeated unnecessarily.
• A human may need to approve an operation later.
• The agent interacts with external systems that may fail independently.
• Multiple workers or services participate in one business process.

For a short chat response, a regular AI SDK agent may be sufficient.

For a multi-hour research workflow or a business operation that cannot be safely restarted from the beginning, durable execution becomes much more important.

The correct architecture is the simplest one that provides the required failure guarantees.

Conclusion

AI agents are becoming more capable, but greater capability introduces additional operational complexity.

An agent that can call tools, delegate tasks, and interact with external systems must also be able to recover from failures, preserve execution progress, and handle interruptions safely.

Durable execution provides the foundation for this.

Checkpointing preserves progress. Retries recover from transient failures. Idempotency prevents duplicate side effects. Durable waits enable human approval without keeping workers occupied. Workflow versioning allows long-running executions to survive application changes.

For AI engineering teams, the goal should not be to make every agent use a workflow engine.

It should be to identify which operations require durability and apply the right execution model to them.

My recommendation: Start with one multi-step workflow that includes an external API call and a human approval step. Implement it with a durable execution framework, simulate process failures, and verify that completed work is not repeated and external side effects are not duplicated.

A production-ready agent is not one that only works when everything goes right.

It is one that knows how to recover when things go wrong.

References and Further Reading

Vercel AI SDK and Workflow

1. Vercel — What is WorkflowAgent? — Durable AI agents, retries, approvals, and resumable execution.
2. Vercel — Build AI Agents with AI SDK and AI Gateway — AI SDK and Workflow integration.
3. Vercel — Workflow SDK — Durable functions, steps, and execution model.
4. Vercel — Migrate from DurableAgent to WorkflowAgent — AI SDK v7 integration changes.

LangGraph

1. LangGraph — Persistence — Checkpointers, stores, state, and memory.
2. LangGraph — Interrupts — Human-in-the-loop and resumable execution.
3. LangGraph — Fault Tolerance — Recovery and execution reliability.

Temporal

1. Temporal — Durable AI — Durable execution for AI applications.
2. Temporal — TypeScript SDK — Workflows, Activities, Workers, and message passing.
3. Temporal — Workflow Determinism — Replay-safe workflow implementation.
4. Temporal — AI Cookbook — AI integrations, agent loops, tool calling, and human approval.

Inngest

1. Inngest — Durable Execution — Steps, retries, checkpoints, and event handling.
2. Inngest — Durable Agents — Agent loops, independent retries, and subagent delegation.
3. Inngest — Human-in-the-Loop — Durable approval and resume patterns.

DBOS

1. DBOS — TypeScript Workflows — Durable workflows and automatic recovery.
2. DBOS — Workflows and Steps — Retry configuration and step execution.
3. DBOS — Architecture — Checkpointing, recovery, and deterministic execution.

Your service handles 500 requests per second.

The CPU is not fully utilized. The database is responding normally. The average processing time looks reasonable.

Yet your p99 latency keeps increasing.

What is happening?

In many cases, the problem isn’t how long requests take to execute. It’s how long they spend waiting before execution even begins.

This is where queueing theory becomes useful.

Queueing theory gives us a way to reason about arrival rates, service capacity, concurrency, waiting time, and system saturation. It helps explain why a service can look healthy under moderate traffic and suddenly experience a dramatic increase in latency as it approaches its capacity.

In this article, we’ll explore the fundamentals of queueing theory and apply them to practical Go backend systems.

We’ll cover:

• Arrival rate, service rate, and utilization
• Little’s Law and its practical applications
• Why latency increases near saturation
• The relationship between concurrency and throughput
• Building bounded queues and worker pools in Go
• Queue sizing, admission control, and overload behavior
• Metrics and load-testing strategies

The goal isn’t to turn every backend engineer into a mathematician. It’s to make better engineering decisions using a few surprisingly powerful relationships.

1. Every Backend Service Is a Queueing System

Consider a typical HTTP service:

Client Requests
      |
      v
   HTTP Server
      |
      v
  Request Queue
      |
      v
  Worker Pool
      |
      v
 Database / API
      |
      v
   Response

Every request follows a similar lifecycle:

1. It arrives at the system.
2. It waits for an available resource.
3. It gets processed.
4. It leaves the system.

The important distinction is that processing time and waiting time are different things.

Imagine two requests:

RequestWaiting timeProcessing timeTotalA5 ms45 ms50 msB250 ms45 ms295 ms

Both requests require exactly the same processing time.

But Request B takes almost six times longer because it spends most of its lifetime waiting.

This is why optimizing database queries or application code alone doesn’t always improve end-to-end latency.

Sometimes the queue is the bottleneck.

2. The Three Variables That Explain Queueing Behavior

Most basic queueing models start with three variables.

Arrival Rate (λ)

The arrival rate represents how frequently new requests enter the system.

For example:

λ = 500 requests/second

It describes incoming demand, not necessarily completed work.

Service Rate (μ)

The service rate describes how quickly a worker can process requests.

If a worker takes an average of 20 milliseconds per request:

μ = 1 / 0.020 = 50 requests/second

One worker can theoretically process 50 requests per second under that average service time.

With 10 equivalent workers, the theoretical aggregate service capacity becomes:

c × μ = 10 × 50 = 500 requests/second

Here, c represents the number of workers.

This assumes the workers can operate independently and that another shared resource doesn’t become the bottleneck.

Utilization (ρ)

Utilization describes how much of the available service capacity is being consumed.

For a simple system with c equivalent workers:

ρ = λ / (c × μ)

Suppose:

• Arrival rate: 400 requests/second
• Service rate per worker: 50 requests/second
• Number of workers: 10

Then:

ρ = 400 / (10 × 50) = 0.8 = 80%

The system is operating at 80% utilization.

That sounds reasonable.

But here’s the important part: 80% utilization does not mean 20% of the system is freely available at all times.

Traffic is variable. Requests don’t arrive at perfectly regular intervals, and processing times aren’t identical.

A system can experience queueing delays even when its average utilization is below 100%.

And as utilization approaches capacity, those delays can grow rapidly.

3. Little’s Law: The Most Useful Queueing Relationship

One of the most practical results in queueing theory is Little’s Law:

L = λ × W

Where:

• L = average number of requests in the entire system
• λ = average arrival rate
• W = average time spent in the entire system

In backend engineering, this gives us a direct relationship between throughput, latency, and concurrency.

For example:

Throughput: 200 requests/second
Average latency: 150 ms

Convert the latency to seconds:

W = 0.150 seconds

Then:

L = 200 × 0.150 = 30

On average, there are approximately 30 requests in the system.

This is incredibly useful because it allows us to estimate concurrency from production measurements.

Let’s look at another example.

A service processes 1,000 requests per second, with an average time in the system of 80 milliseconds.

L = 1,000 × 0.080 = 80

The service has approximately 80 requests in flight on average.

This doesn’t mean we should automatically configure 80 workers.

It means the observed workload has an average concurrency of approximately 80. Worker count, resource limits, queue capacity, and service time still need to be considered separately.

Queue length versus requests in the system

There’s an important distinction here.

Little’s Law can describe either the entire system or just the waiting queue, provided the measurements use matching definitions.

For the entire system:

L = λ × W

For the waiting queue:

Lq = λ × Wq

Where:

• Lq = average number of requests waiting in the queue
• λ = average arrival rate into the queue
• Wq = average time spent waiting in the queue

The distinction matters when instrumenting a Go worker pool.

If we measure only queue depth, we shouldn’t compare it directly with total request latency. We need to compare it with queue waiting time.

When should you use Little’s Law?

It’s useful for:

• Estimating average in-flight requests
• Understanding queue growth
• Planning concurrency limits
• Checking whether observed metrics are consistent
• Estimating the waiting time implied by a queue
• Identifying where work accumulates

However, Little’s Law describes long-run averages in a stable system. It doesn’t predict p99 latency or tell us how a system behaves during a sudden traffic spike.

For that, we need to understand utilization and variability.

4. Why Latency Explodes Near Saturation

Let’s consider a simplified single-server queue.

Assume:

• Requests arrive according to a Poisson process.
• Service times are exponentially distributed.
• There is one server.
• The queue has unlimited capacity.

This is commonly called an M/M/1 queue.

For this particular model, the expected time in the system is:

W = 1 / (μ — λ)

And the expected waiting time in the queue is:

Wq = ρ / (μ — λ)

These formulas are model-specific. Real production systems often have multiple workers, correlated traffic, variable service times, and bounded queues.

But the model illustrates an important behavior.

Assume the server can process 100 requests per second.

Arrival rateUtilizationAverage time in system50 req/s50%20 ms70 req/s70%33.3 ms80 req/s80%50 ms90 req/s90%100 ms95 req/s95%200 ms99 req/s99%1,000 ms

Illustrative M/M/1 model, with a service rate of 100 requests per second.

The service rate never changes.

Yet the average response time increases from 20 milliseconds to one full second as utilization approaches 100%.

This is the queueing effect that makes operating a service continuously at maximum utilization dangerous.

The system doesn’t necessarily become slower at processing individual requests. It becomes slower at getting requests to the point where processing can begin.

This also explains why a small traffic increase can cause a disproportionately large latency increase.

In real systems, variability can make the situation worse. Bursty arrivals, slow dependencies, garbage collection pauses, lock contention, and uneven request costs all contribute to queueing.

High utilization leaves very little room to absorb variability.

5. Concurrency Is Not the Same as Throughput

A common approach to improving performance is increasing concurrency.

If 10 workers aren’t enough, why not use 100?

Because more concurrency doesn’t necessarily produce more throughput.

Imagine a service that queries a database.

Go Service
    |
    v
100 Concurrent Requests
    |
    v
Database Connection Pool
    |
    v
20 Available Connections

Only 20 requests can use a database connection simultaneously.

The remaining requests may wait for a connection.

Increasing application concurrency from 100 to 500 doesn’t increase database capacity.

It may simply move the queue from the application into the database connection pool.

This is an important principle:

Increasing concurrency can move the bottleneck without removing it.

A useful way to reason about concurrency is Little’s Law:

L = λ × W

If throughput remains constant while the number of concurrent requests increases, the average time spent in the system must also increase.

For example:

Throughput: 100 requests/second
Concurrency: 100
Average time: 1 second

Now suppose concurrency increases to 300, but throughput remains 100 requests per second.

Concurrency: 300
Average time: 3 seconds

The system is carrying more work, but it isn’t completing more work.

This is why measuring concurrency without measuring throughput and latency can be misleading.

6. Implementing a Bounded Worker Pool in Go

Let’s move from theory to implementation.

A worker pool is a common way to control how much work executes concurrently.

But a worker pool alone isn’t enough.

If the incoming queue is unbounded, the service can continue accepting work long after its processing capacity is exhausted.

A bounded queue provides an explicit limit.

Here’s a simplified implementation using Go channels:

package main

import (
    "context"
    "errors"
    "sync"
)

var (
    ErrQueueFull  = errors.New("worker queue is full")
    ErrPoolClosed = errors.New("worker pool is closed")
)

type Job func(context.Context) error

type Pool struct {
    jobs chan Job
    ctx    context.Context
    cancel context.CancelFunc
    mu     sync.Mutex
    closed bool
    wg sync.WaitGroup
}

func NewPool(
    parent context.Context,
    workers int,
    queueSize int,
) *Pool {
    if workers <= 0 {
        panic("workers must be greater than zero")
    }
    if queueSize < 0 {
        panic("queue size cannot be negative")
    }
    ctx, cancel := context.WithCancel(parent)
    p := &Pool{
        jobs:   make(chan Job, queueSize),
        ctx:    ctx,
        cancel: cancel,
    }
    p.wg.Add(workers)
    for i := 0; i < workers; i++ {
        go p.worker()
    }
    return p
}

func (p *Pool) Submit(job Job) error {
    if job == nil {
        return errors.New("job cannot be nil")
    }
    p.mu.Lock()
    defer p.mu.Unlock()
    if p.closed {
        return ErrPoolClosed
    }
    if err := p.ctx.Err(); err != nil {
        return err
    }
    select {
    case p.jobs <- job:
        return nil
    default:
        return ErrQueueFull
    }
}

func (p *Pool) worker() {
    defer p.wg.Done()
    for {
        select {
        case <-p.ctx.Done():
            return
        case job := <-p.jobs:
            if p.ctx.Err() != nil {
                return
            }
            if job != nil {
                _ = job(p.ctx)
            }
        }
    }
}

func (p *Pool) Close() {
    p.mu.Lock()
    if p.closed {
        p.mu.Unlock()
        return
    }
    p.closed = true
    p.cancel()
    p.mu.Unlock()
    p.wg.Wait()
}

The implementation demonstrates a few important ideas:

• The number of workers is fixed.
• The queue has a bounded capacity.
• Submit rejects work immediately when the queue is full.
• Worker execution receives a cancellation context.
• Submission and shutdown are coordinated using a mutex.
• Close cancels workers and waits for them to exit.

This is an intentionally small example, not a complete production job-processing framework.

In particular, it uses immediate cancellation rather than graceful draining. It also doesn’t provide per-job results, panic recovery, retry handling, or error reporting.

Those behaviors should be designed explicitly for the workload.

Also remember that context cancellation is cooperative. A job that ignores its context can prevent Close from returning.

Why bounded queues matter

Consider a service with:

Workers: 20
Queue capacity: 500
Average processing time: 100 ms

Each worker can theoretically process:

μ = 1 / 0.1 = 10 jobs/second

Aggregate service capacity:

c × μ = 20 × 10 = 200 jobs/second

If all workers are busy and the queue contains 500 jobs, the approximate queue-draining time is:

Wq ≈ 500 / 200 = 2.5 seconds

This is a rough estimate assuming the workers continue processing at the same aggregate rate and no new jobs arrive.

If your API has a 500-millisecond response deadline, accepting work into this queue may be pointless.

The request might time out before a worker even starts processing it.

A queue is not automatically useful just because it can hold more work.

It should hold only the amount of work your system can reasonably process before that work becomes irrelevant.

7. Queue Capacity Is a Latency Decision

Developers often select queue capacity using arbitrary values:

make(chan Job, 1000)

But why 1,000?

Why not 100 or 10,000?

A better approach is to start with a latency budget.

Suppose:

Maximum acceptable queue waiting time: 500 ms
Aggregate processing capacity: 200 jobs/second

A rough initial queue capacity can be estimated as:

Queue Capacity ≈ Aggregate Service Capacity × Target Queue Waiting Time

Given:

• Aggregate service capacity: 200 jobs/second
• Target queue waiting time: 0.5 seconds

The calculation becomes:

Q ≈ 200 × 0.5 = 100 jobs

This suggests that a queue capacity around 100 jobs corresponds to approximately 500 milliseconds of waiting at the assumed completion rate, if the queue is full and no new jobs arrive.

This is not a worst-case guarantee.

Actual waiting time depends on worker availability, service-time distribution, traffic bursts, and the number of jobs already executing.

Still, it gives us a much more defensible starting point than choosing a large number without understanding its consequences.

Queue age is often more useful than queue depth

Imagine two systems:

System A:
Queue depth: 100
Oldest job: 10 ms

System B:
Queue depth: 20
Oldest job: 2 seconds

System B has fewer queued jobs, but its users may be experiencing a much worse situation.

Queue depth tells us how much work is waiting.

Queue age tells us how long that work has already been waiting.

For latency-sensitive systems, both metrics are important.

8. Admission Control: Decide Whether Work Should Enter

A bounded queue provides a capacity limit.

Admission control adds a policy for deciding whether incoming work should be accepted.

For example:

func HandleRequest(
    ctx context.Context,
    queue chan Job,
    job Job,
) error {
    if err := ctx.Err(); err != nil {
        return err
    }

    select {
    case queue <- job:
        return nil
    case <-ctx.Done():
        return ctx.Err()
    default:
        return ErrQueueFull
    }
}

This is a simplified non-blocking admission decision.

When the queue is full, the request is rejected instead of waiting indefinitely.

In a real HTTP handler, this could translate into an overload response such as:

HTTP/1.1 503 Service Unavailable
Retry-After: 1

The exact status and retry policy depend on the API contract.

There are several possible admission strategies:

Reject immediately

Useful when waiting is unlikely to produce a result within the request deadline.

Wait for a bounded period

Useful when short bursts are expected and a small amount of waiting is acceptable.

Prioritize work

Useful when critical operations should be processed ahead of background or low-priority tasks.

Apply load shedding

Useful when the system is overloaded and preserving the health of the remaining workload is more important than accepting every request.

The important part is to make overload behavior intentional.

9. The Connection to Backpressure, Load Shedding, and Adaptive Concurrency

Queueing theory provides the foundation for several resilience patterns we’ve discussed in this series.

They solve different parts of the same problem.

PatternWhat it controlsBackpressureHow quickly work enters the systemLoad sheddingWhich work should be rejectedBulkheadsHow resources are isolatedRetry budgetsHow much additional traffic retries generateAdaptive concurrencyHow many requests execute simultaneouslyHedged requestsHow to reduce tail latency through controlled parallelism

These mechanisms become much more effective when we understand the queueing behavior behind them.

For example, an adaptive concurrency controller can reduce the number of in-flight requests when latency indicates that a dependency is approaching saturation.

A bounded queue prevents the service from accumulating unlimited waiting work.

Load shedding rejects requests that cannot be processed within a useful time window.

Together, these mechanisms help keep the system stable under changing traffic conditions.

10. What Should You Measure in Production?

Queueing theory is useful only when its variables can be observed.

At a minimum, I’d monitor the following metrics.

MetricWhy it mattersArrival rateIncoming demandCompletion rateActual processing capacityQueue depthAmount of waiting workQueue ageHow long work has been waitingIn-flight requestsCurrent concurrencyQueue wait durationTime spent waiting before executionService durationActual processing timeRejected requestsAdmission-control behaviorp95 / p99 latencyUser-facing latency distribution

A particularly useful metric is the difference between total request duration and processing duration.

Total request duration: 350 ms
Queue wait:              250 ms
Processing duration:     100 ms

If processing duration is stable but total latency increases, the queue is a strong candidate for investigation.

For Go services, you can expose these measurements through Prometheus.

Example metric names:

http_requests_total
http_request_duration_seconds

worker_pool_queue_depth
worker_pool_queue_wait_seconds
worker_pool_active_workers
worker_pool_jobs_rejected_total
worker_pool_job_duration_seconds

Use histograms for durations and counters for events. Queue depth and active workers are gauges.

Avoid putting unbounded values such as user IDs, request IDs, or raw URLs into metric labels. That creates unnecessary cardinality.

A practical diagnostic example

Suppose a production service starts showing the following behavior:

Arrival rate:          800 req/s
Completion rate:       790 req/s
Active workers:        100
Queue depth:           Increasing
Queue wait p95:        450 ms
Processing p95:        60 ms
HTTP latency p95:      520 ms

The first instinct might be to optimize the handler.

But the measurements tell a different story.

Processing time is relatively low compared with total latency.

The queue is accumulating work, and the service is completing slightly fewer requests than it receives.

Adding more workers may help only if the downstream resources have spare capacity.

Before increasing concurrency, investigate:

• Database connection pool saturation
• Downstream API latency
• CPU and memory pressure
• Lock contention
• Uneven job execution times
• Whether requests are arriving in bursts

The goal is to identify the actual bottleneck instead of simply increasing the number of workers.

11. Testing Queueing Behavior

A worker pool that works correctly with 10 requests may behave very differently with 10,000.

Load tests should examine more than maximum throughput.

I’d test at least four scenarios.

1. Normal load

Run below expected capacity and establish baseline latency, throughput, and resource usage.

2. Near-saturation load

Gradually increase arrival rate toward service capacity. Observe how queue wait and p99 latency change.

3. Burst traffic

Send a large number of requests in a short interval. Verify that the queue remains bounded and overload behavior is predictable.

4. Dependency slowdown

Artificially increase downstream service time. Observe whether concurrency, queue depth, and rejection rates remain controlled.

For a Go worker pool, also test:

• Queue-full behavior
• Cancellation before execution
• Shutdown while workers are active
• Jobs that ignore context cancellation
• Worker panics, if panic recovery is part of the design
• Concurrent submission during shutdown

A particularly important test is whether your service continues accepting work after its useful processing capacity has been exhausted.

A successful load test isn’t necessarily one where every request is accepted.

Sometimes the better result is a system that rejects excess work quickly while keeping accepted requests within their latency objectives.

12. A Practical Checklist

Before deploying a queue-based Go service, ask:

• Is the queue bounded?
• Is the queue capacity based on a latency budget?
• Do we measure queue wait time separately from processing time?
• Can requests be rejected when the system is saturated?
• Do jobs respect context cancellation?
• Can shutdown stop workers without leaking goroutines?
• Are downstream connection pools and concurrency limits understood?
• Do we monitor queue depth and oldest-job age?
• Have we tested burst traffic and dependency slowdown?
• Are we optimizing useful throughput rather than simply maximizing accepted requests?

These questions help turn queueing theory into concrete engineering decisions.

Conclusion

Queueing theory explains something that’s easy to overlook when building backend services:

A request doesn’t need to be expensive to become slow. It only needs to wait.

Little’s Law helps us reason about concurrency and latency. Utilization explains why waiting time increases rapidly near capacity. Bounded queues and admission control give us ways to prevent uncontrolled accumulation of work.

In Go, goroutines and channels make it relatively easy to build concurrent systems. But concurrency itself isn’t a capacity-planning strategy.

The important questions are:

• How much work enters the system?
• How quickly can the system complete it?
• How much work can wait?
• How long can that work remain useful?
• What happens when demand exceeds capacity?

A reliable service doesn’t need to accept every request.

It needs to process accepted work predictably, protect its dependencies, and fail fast when it can no longer meet its latency objectives.

The goal isn’t to keep every worker busy. It’s to keep the system healthy and useful under load.

References

• Little’s Law as Viewed on its 50th Anniversary — INFORMS
• MIT — Queueing Models and Little’s Law
• Amdahl’s Law for Tail Latency — Communications of the ACM
• Go Documentation — Context
• Go Documentation — Go Memory Model

How to understand, monitor, debug, and optimize agentic AI systems using OpenTelemetry and modern AI observability practices.

1. Why Is Observability Different for AI Agents?

Traditional backend observability is built around relatively predictable execution paths.

A request enters an application, passes through several services, interacts with a database, and eventually returns a response. We can use logs, metrics, and distributed traces to understand what happened and where a problem occurred.

AI agents introduce a different level of complexity.

An agent does not necessarily follow a predefined execution path. It may call an LLM multiple times, select tools dynamically, retry failed operations, delegate tasks to other agents, retrieve external context, or wait for additional user input.

Consider a relatively simple task:

A user asks an AI assistant to analyze a merge request and identify potential problems.

Behind the scenes, the application might:

1. Receive the user’s request.
2. Invoke a coordinator agent.
3. Call an LLM to determine the next action.
4. Retrieve the merge request from GitLab.
5. Read the changed files.
6. Delegate security analysis to another agent.
7. Execute additional tool calls.
8. Evaluate the collected results.
9. Generate a final report.

The user sees one response. The application has actually executed a multi-step workflow involving several components.

Now imagine the response takes 45 seconds instead of the expected 8 seconds.

Where should we investigate?

• Did the model take too long to respond?
• Did a tool call time out?
• Was the same operation executed repeatedly?
• Did the agent generate unnecessary steps?
• Did a subagent introduce additional latency?
• Was too much context sent to the model?
• Did a retry increase the overall execution time and cost?

Traditional request-level logging may tell us that the endpoint took 45 seconds. It does not necessarily explain why.

AI observability is about understanding the entire execution path, not just measuring the final API response.

This is where OpenTelemetry becomes particularly useful.

2. What Is OpenTelemetry?

<escape>**</escape><Entity value=”OpenTelemetry (OTel)” category=”software” disambig=”Open-source observability framework”/> is an open-source observability framework for collecting, processing, and exporting telemetry data.

It provides a vendor-neutral instrumentation model that allows applications to generate telemetry and send it to different observability platforms.

Instead of implementing a separate monitoring integration for every vendor, we can instrument an application using OpenTelemetry and choose where the telemetry should be exported.

OpenTelemetry is built around three primary signals:

SignalWhat it tells usAI agent exampleTracesHow an operation was executedModel call → tool → subagentMetricsHow the system performs over timeToken usage, latency, error rateLogsWhat happened at a specific pointTool timeout or agent exception

These signals complement each other.

A metric can tell us that the average execution time has increased. A trace can identify which operation introduced the delay. A log can provide the error details associated with that operation.

OpenTelemetry also provides context propagation, APIs, SDKs, semantic conventions, and the OpenTelemetry Collector.

An important implementation detail: as of November 2026, the OpenTelemetry JavaScript documentation lists traces and metrics as Stable, while logs remain in Development. The maturity of a specific GenAI convention should also be evaluated separately from the maturity of the underlying OTel signal.

References: OpenTelemetry Documentation · OpenTelemetry JavaScript

3. Understanding Distributed Tracing

Before discussing AI-specific instrumentation, we need to understand traces and spans.

A trace represents the complete execution path of an operation.

A span represents one individual unit of work within that execution.

For example:

Trace: POST /api/agent
|
+-- HTTP Request
    |
    +-- Coordinator Agent
        |
        +-- LLM Generation
        |
        +-- Tool: GitLab - Get Merge Request
        |
        +-- Tool: GitLab - Get Changed Files
        |
        +-- Security Agent
        |   |
        |   +-- LLM Generation
        |   |
        |   +-- Tool: Security Analysis
        |
        +-- Testing Agent
        |   |
        |   +-- LLM Generation
        |   |
        |   +-- Tool: Test Coverage
        |
        +-- Final Response Generation

Each span has information such as:

• Trace ID
• Span ID
• Parent span ID
• Start and end timestamps
• Duration
• Attributes
• Events
• Status and exception information

This creates a hierarchical representation of the complete execution.

The key benefit is that we can move from a high-level workflow to an individual model call or tool execution without losing the relationship between them.

For AI agents, this distinction is important because one user request can generate many independent operations.

Why is context propagation important?

In distributed systems, a trace often crosses multiple services.

Imagine a Next.js application calling a .NET backend, which then delegates a task to a Python agent and receives a result from another service.

Without context propagation, every service may generate its own independent trace.

We would see several unrelated operations instead of one complete workflow.

OpenTelemetry uses W3C Trace Context as its standard propagation format. The traceparent header carries trace and parent-span information between instrumented services.

The same principle applies to agent-based architectures.

When an agent delegates a task to another agent through A2A, we should preserve the correlation between the originating request and the remote execution.

Automatic HTTP instrumentation can handle propagation when the HTTP instrumentation and context manager are configured correctly. Custom transports and asynchronous message queues may require explicit context injection and extraction.

The important point is that protocol-level interoperability and observability context propagation are separate concerns. Using A2A does not, by itself, guarantee that all distributed spans will appear in one trace.

References: OpenTelemetry Traces · Context Propagation · W3C Trace Context

4. OpenTelemetry and GenAI Semantic Conventions

One of the challenges in AI observability is the lack of consistent naming and structure across different AI frameworks and providers.

For example, one framework might represent an LLM request as a generic operation, while another might expose model-specific attributes.

This makes it difficult to compare different implementations using the same monitoring platform.

OpenTelemetry addresses this through Semantic Conventions.

Semantic conventions define common attribute names, operation types, and instrumentation expectations so that telemetry from different implementations can be interpreted consistently.

The GenAI semantic conventions extend this approach to generative AI applications.

Some important attributes include:

AttributePurposegen_ai.provider.nameIdentifies the model providergen_ai.request.modelIdentifies the requested modelgen_ai.response.modelIdentifies the actual response modelgen_ai.operation.nameDescribes the operationgen_ai.usage.input_tokensInput token usagegen_ai.usage.output_tokensOutput token usagegen_ai.conversation.idAssociates operations with a conversationgen_ai.agent.nameIdentifies the agentgen_ai.tool.nameIdentifies the executed tool

For example, an LLM invocation could be represented by:

{
  "gen_ai.operation.name": "chat",
  "gen_ai.provider.name": "openai",
  "gen_ai.request.model": "example-model",
  "gen_ai.usage.input_tokens": 1200,
  "gen_ai.usage.output_tokens": 350
}

This is a simplified span attribute example.

The benefit is that observability platforms do not need to understand a different custom schema for every application.

Agent-specific conventions

The GenAI semantic conventions also define operations for agents and workflows.

Some examples include:

• invoke_agent
• invoke_workflow
• execute_tool
• plan
• retrieval

This is particularly valuable for multi-agent architectures.

We can distinguish an individual model request from an entire agent invocation or a workflow coordinating multiple agents.

For example:

invoke_workflow merge-request-review
|
+-- invoke_agent coordinator
|   |
|   +-- chat model
|   +-- execute_tool fetch_merge_request
|
+-- invoke_agent security-agent
|   |
|   +-- chat model
|   +-- execute_tool analyze_vulnerabilities
|
+-- invoke_agent testing-agent
|   |
|   +-- chat model
|   +-- execute_tool analyze_test_coverage
|
+-- Final Aggregation

This makes the execution model much easier to understand.

An important maturity consideration: As of November 2, 2026, the OpenTelemetry GenAI agent spans and GenAI metrics documents are still marked Development. These conventions are valuable design references, but teams should pin their instrumentation versions and avoid treating every proposed attribute or metric as a stable, universally implemented contract.

The current convention documents include:

• Semantic Conventions for GenAI Spans
• Semantic Conventions for Agent Spans
• Semantic Conventions for GenAI Metrics

5. Integrating OpenTelemetry with Vercel AI SDK

Since many TypeScript applications use the Vercel AI SDK to implement model interactions and agent workflows, it is worth looking at its native telemetry integration.

The AI SDK already supports OpenTelemetry-based instrumentation.

It can generate spans for:

• Text generation
• Streaming generation
• Individual model calls
• Agent execution
• Tool execution
• Embedding operations
• Reranking

This means we do not need to manually instrument every model call and tool execution just to obtain basic execution visibility.

Step 1: Install the packages

For an AI SDK v7 application:

pnpm add @ai-sdk/otel @vercel/otel

The OpenTelemetry integration is provided through @ai-sdk/otel.

This is an important version-specific detail.

In AI SDK v7, OpenTelemetry integration moved out of the core ai package. Older examples using the v6 experimental_telemetry configuration should not be copied directly into a v7 application.

Step 2: Register telemetry in Next.js

Create an instrumentation.ts file in the project root.

import { registerOTel } from '@vercel/otel';
import { registerTelemetry } from 'ai';
import { OpenTelemetry } from '@ai-sdk/otel';

const serviceName = 'ai-agent-platform'

export function register() {

  registerOTel({
    serviceName,
  });

  registerTelemetry(
    new OpenTelemetry({
      recordInputs: false,
      recordOutputs: false,
    }),
  );
}

There are two responsibilities here:

• registerOTel initializes the OpenTelemetry infrastructure for the Next.js application.
• registerTelemetry registers the AI SDK-specific integration that creates GenAI spans.

I have disabled input and output capture in this example intentionally.

AI SDK telemetry can record prompts and generated outputs by default. In an enterprise environment, capturing this data requires an explicit privacy and security decision.

We will return to this in the security section.

For a regular Node.js application without Next.js, register the OpenTelemetry integration once at application startup.

Step 3: Instrument AI SDK calls

Once telemetry is registered, supported AI SDK operations automatically emit telemetry through the registered integration.

We can also provide a functionId to identify the business operation or agent.

import { generateText } from 'ai';

const result = await generateText({
  model: myModel,

  instructions: `
    You are a financial analysis assistant.
    Analyze the supplied information carefully.
  `,

  prompt: 'Analyze the latest portfolio performance.',

  telemetry: {
    functionId: 'portfolio-analysis-agent',
  },
});

The functionId helps distinguish operations in traces.

For example, rather than seeing only generic model invocation spans, we can identify a specific business operation such as:

invoke_agent portfolio-analysis-agent

Step 4: Understand the generated spans

The AI SDK v7 OpenTelemetry integration follows the GenAI semantic conventions.

For generateText and streamText, the integration records three main types of spans:

1. Agent invocation span

The root span represents the complete generation operation, including its execution steps and tool calls.

Example:

invoke_agent portfolio-analysis-agent

2. Step and model-call spans

Each model interaction is represented by step and provider-call spans.

These allow us to understand how many model calls occurred during an operation and how long each one took.

3. Tool execution spans

Each client-side tool execution receives its own span.

For example:

invoke_agent portfolio-analysis-agent
|
+-- step 0
|   |
|   +-- chat example-model
|   +-- execute_tool get_portfolio
|
+-- step 1
|   |
|   +-- chat example-model
|   +-- execute_tool get_market_data
|
+-- step 2
    |
    +-- chat example-model

The resulting trace provides a much more complete representation of an agent workflow than a single model request log.

The integration also records usage information such as input tokens, output tokens, cache usage, model identifiers, and operation duration when these values are available.

Step 5: Add additional application context

Sometimes the standard GenAI attributes are not enough.

For example, we may want to know which business workflow initiated an operation or which application component owns it.

The AI SDK provides enrichSpan for adding custom attributes:

import { registerTelemetry } from 'ai';
import { OpenTelemetry } from '@ai-sdk/otel';

registerTelemetry(
  new OpenTelemetry({
    recordInputs: false,
    recordOutputs: false,
    enrichSpan: ({ spanType, operationId }) => ({
      'app.component': 'portfolio-assistant',
      'app.operation_id': operationId,
      'app.span_type': spanType,
    }),
  }),
);

The exact context available to enrichSpan depends on the span and the SDK's integration API.

The objective is to enrich the trace with useful application-level context without copying sensitive business data into telemetry.

References: AI SDK Telemetry · Next.js OpenTelemetry · AI SDK v7 Migration Guide

6. Instrumenting Your Own Agent Runtime

Built-in instrumentation is useful, but it does not automatically understand every application-level decision.

For example, an AI SDK trace may show model calls and tool executions, but it may not tell us why a coordinator selected a particular specialist agent or how long a business workflow spent waiting for an approval.

These are application-level operations.

We should instrument them explicitly.

OpenTelemetry provides a tracing API for creating custom spans.

Consider a custom agent orchestration function:

import {
  trace,
  SpanStatusCode,
} from '@opentelemetry/api';

const tracer = trace.getTracer('agent-orchestrator');

export async function runReviewWorkflow() {
  return tracer.startActiveSpan(
    'workflow.merge-request-review',
    async (span) => {
      span.setAttribute(
        'workflow.name',
        'merge-request-review',
      );
      span.setAttribute(
        'workflow.version',
        '1.0.0',
      );

      try {
        const result = await executeReview();
        span.setStatus({
          code: SpanStatusCode.OK,
        });
        return result;
      } catch (cause) {
        const error =
          cause instanceof Error
            ? cause
            : new Error(String(cause));
        span.recordException(error);
        span.setStatus({
          code: SpanStatusCode.ERROR,
        });
        throw error;
      } finally {
        span.end();
      }

    },
  );
}

Any supported AI SDK operations executed inside this active span can become child operations in the same trace, provided the OpenTelemetry context is correctly propagated.

We now have two levels of observability:

• Framework-level instrumentation describes model and tool execution.
• Application-level instrumentation describes business workflows and orchestration decisions.

This distinction is important.

We should not expect a model instrumentation library to understand the complete business meaning of an application request.

Instrumenting remote agent calls

For A2A-based systems, distributed tracing becomes even more important.

Imagine a coordinator running in one service and a security agent running in another service.

The coordinator should create a child operation for the delegation. The remote service should continue the trace when the execution context is propagated.

Conceptually:

Trace: merge-request-review
|
+-- Coordinator Agent
|   |
|   +-- Model Call
|   |
|   +-- A2A: Delegate to Security Agent
|       |
|       +-- Security Agent
|           |
|           +-- Model Call
|           |
|           +-- Tool Execution
|
+-- Final Aggregation

The HTTP request must carry the appropriate trace context, typically through W3C traceparent and tracestate headers.

For custom transports or asynchronous queues, explicitly inject and extract the OpenTelemetry context.

This is how we avoid fragmented traces across independently deployed agents.

7. What Should We Measure?

Collecting telemetry is not enough.

We need to define which metrics are useful for understanding the health and quality of an agent platform.

I would divide AI observability metrics into five categories.

7.1. Latency Metrics

These describe how long operations take.

MetricWhy it mattersEnd-to-end workflow durationMeasures total user-facing execution timeModel request durationIdentifies slow inference callsTime to first tokenMeasures perceived response latencyTool execution durationIdentifies slow external operationsAgent invocation durationMeasures the execution time of one agentQueue wait durationIdentifies scheduling bottlenecks

For streaming applications, time to first token is particularly useful.

A workflow that takes 20 seconds but starts responding after 300 ms may provide a better experience than one that takes 10 seconds and remains silent for the first 8 seconds.

7.2. Reliability Metrics

These help us understand whether workflows complete successfully.

Examples:

• Agent execution success rate
• Tool execution failure rate
• Model request failure rate
• Timeout rate
• Retry rate
• Task cancellation rate
• Human intervention rate
• Incomplete workflow rate

A high retry rate can be an early indicator of a problematic integration or an unstable external dependency.

We should also distinguish between a successful model response and a successful business workflow.

An LLM can return a perfectly valid response while the overall task fails to achieve its objective.

7.3. Token and Model Usage

Token usage should be tracked at multiple levels.

MetricDescriptionInput tokensTokens sent to the modelOutput tokensTokens generated by the modelCached input tokensTokens served from provider cacheReasoning tokensModel-reported reasoning token usageTokens per workflowTotal usage across an entire taskTokens per agentUsage attributed to an individual agent

For multi-step workflows, we should aggregate usage from all model calls rather than looking only at the final response.

7.4. Agent and Tool Behavior

These metrics describe how efficiently agents perform their work.

Examples:

• Number of tool calls per task
• Number of model calls per agent invocation
• Number of agent handoffs
• Repeated tool-call frequency
• Tool-selection success
• Average steps per workflow
• Average number of clarification requests

An agent repeatedly calling the same tool can indicate a reasoning loop, an incorrect tool result, or an orchestration problem.

The trace helps us identify the execution pattern. Metrics help us understand how frequently the pattern occurs.

7.5. Quality Metrics

This is where AI observability starts to overlap with AI evaluation.

Examples:

• Answer correctness
• Groundedness
• Retrieval relevance
• Tool-selection accuracy
• Task completion
• Policy adherence
• Human acceptance rate

These metrics cannot always be derived from execution telemetry alone.

We need an evaluation strategy to measure the actual quality of the result.

8. Cost Control: Understanding the Real Cost of an Agent

One of the biggest mistakes in AI platforms is measuring only model token usage and treating it as the total cost of the workflow.

An agent-based application may consume resources across several layers:

• LLM inference
• Embedding generation
• Reranking
• External API calls
• Database queries
• Vector database operations
• Container or sandbox execution
• Remote agent invocations
• Infrastructure and networking

The total cost should reflect the entire execution path.

A useful starting point is to define a model cost formula:

[ C{\text{model}} = \sum{i=1}^{n} \left( T{\text{input},i}P{\text{input},i} + T{\text{output},i}P{\text{output},i} \right) ]

Where:

• (T) represents token usage.
• (P) represents the applicable price per token.
• (i) represents an individual model operation.

The pricing model may also need separate rates for cached input, audio, images, reasoning tokens, or other modalities.

We should use the actual provider-reported billable usage wherever available and avoid double-counting cached or reasoning token subsets.

The complete workflow cost can then be represented as:

[ C{\text{workflow}} = C{\text{model}} + C{\text{tools}} + C{\text{compute}} + C_{\text{storage}} ]

This may be an estimate rather than an exact invoice calculation, depending on which provider and infrastructure usage data are available.

Why is cost per successful task more useful?

Imagine two implementations:

MetricAgent AAgent BAverage cost per attempt$0.04$0.07Successful completion rate90%98%Cost per successful task~$0.044~$0.071

Agent A is cheaper per attempt, but Agent B completes more tasks successfully.

Neither is automatically better.

The correct decision depends on the business value of successful completion, quality, latency, and the cost of failed attempts.

A useful metric is:

[ \text{Cost per successful task} = \frac{\text{Total execution cost}} {\text{Number of successful tasks}} ]

The numerator should include failed attempts that consumed resources, rather than counting only successful runs.

Model selection and routing

Observability data can also help us make better model-routing decisions.

For example:

• Use a smaller model for classification or simple extraction.
• Use a more capable model for complex analysis.
• Avoid unnecessary model calls when a deterministic operation is sufficient.
• Reuse cached context when appropriate.
• Set explicit iteration and tool-call limits.
• Stop an agent when a task exceeds its budget.

The important principle is that model selection should be based on measured task quality and total execution cost, not just the price of an individual inference request.

Token metrics in OpenTelemetry

The current GenAI token metric conventions include proposed counters such as:

• gen_ai.client.inference.usage.input_tokens
• gen_ai.client.inference.usage.output_tokens
• gen_ai.client.inference.usage.cache_read.input_tokens
• gen_ai.client.inference.usage.reasoning.output_tokens

These conventions are still in Development.

There is also an important distinction between token usage attributes on individual spans and cumulative token usage metrics.

A span attribute describes an individual operation. A counter or histogram is used to analyze consumption across many operations.

For production cost reporting, maintain an application-level pricing registry with model and provider versions, effective dates, and the relevant unit prices.

This makes historical cost calculations more reliable when provider pricing changes.

References: GenAI Token Metrics · GenAI Metrics

9. Observability Is Not Evaluation

This distinction is worth emphasizing.

Observability tells us what happened during execution.

Evaluation tells us whether the result was good enough.

A trace can show that an agent:

• Called the correct model.
• Used three tools.
• Retrieved five documents.
• Completed within four seconds.
• Returned a response without an exception.

But none of these facts proves that the answer was correct.

Consider a RAG application.

The retrieval operation may complete successfully, and the model may return a fluent answer. However, the retrieved documents may be irrelevant or the final answer may contain unsupported information.

From an operational perspective, the request succeeded.

From a quality perspective, it failed.

We need both observability and evaluation.

Offline Evaluation

Offline evaluation measures an AI application against a predefined dataset.

For example, we can create 100 representative questions and expected outcomes.

Whenever we modify a prompt, model, retrieval strategy, or agent workflow, we run the same dataset again.

This helps identify regressions before deploying changes.

Useful evaluation dimensions include:

• Answer correctness
• Relevance
• Groundedness
• Tool selection
• Expected task completion
• Policy compliance

Online Evaluation

Online evaluation measures actual production interactions.

We can select a sample of production traces and evaluate them using:

• Rule-based checks
• Code-based evaluators
• LLM-as-a-Judge
• Human feedback

The important part is to turn real production failures into future test cases.

For example:

1. A production trace shows that an agent selected the wrong tool.
2. We identify the relevant input and expected behavior.
3. We create a regression dataset entry.
4. We update the prompt or agent implementation.
5. We rerun offline evaluations.
6. We deploy and monitor the next set of production traces.

This creates a continuous improvement loop.

Production Traces
       |
       v
Failure Analysis
       |
       v
Evaluation Dataset
       |
       v
Offline Evaluation
       |
       v
Prompt / Model / Agent Changes
       |
       v
Deployment
       |
       v
Online Evaluation
       |
       +------> New Regression Cases

Tools worth evaluating

Two examples of AI-focused observability and evaluation platforms are:

Langfuse

Provides tracing, evaluations, datasets, experiments, and online evaluation of production traffic. It can help connect execution traces with quality measurements.

Documentation: https://langfuse.com/docs/evaluation/overview

Arize Phoenix

An open-source observability and evaluation platform built around OpenTelemetry-based instrumentation and OpenInference conventions.

It provides tracing, evaluation, datasets, experiments, and debugging capabilities.

Documentation: https://phoenix.arize.com/

These tools complement OpenTelemetry. They do not replace the underlying need for correct instrumentation or a well-defined evaluation strategy.

Reference: Langfuse Evaluation Concepts · Arize Phoenix

10. The Role of OpenTelemetry Collector

In a production architecture, applications should not necessarily export telemetry directly to every observability backend.

This is where the OpenTelemetry Collector becomes useful.

The Collector receives telemetry from instrumented applications, processes it, and exports it to one or more destinations.

It provides a central place for:

• Batching telemetry
• Filtering unwanted data
• Applying processors
• Handling authentication
• Reducing telemetry volume
• Sampling traces
• Routing signals to different backends

A simplified architecture looks like this:

                AI Applications
                       |
            +----------+----------+
            |          |          |
            v          v          v
         Next.js     .NET API   Python Agent
            |          |          |
            +----------+----------+
                       |
                      OTLP
                       |
                       v
             OpenTelemetry Collector
                       |
              +--------+--------+
              |        |        |
              v        v        v
            Traces   Metrics   Logs
              |        |        |
              v        v        v
            Jaeger   Prometheus  Log Backend
              |
              v
            Grafana

The same telemetry can also be routed to AI-specific observability platforms where supported.

Why centralize collection?

In enterprise environments, we may have several applications and teams using different frameworks and languages.

A Collector provides a common ingestion point and makes it possible to apply centralized policies.

For example, a Collector can filter sensitive attributes before telemetry reaches the final backend.

It also allows us to change exporters or observability providers without changing the instrumentation in every application.

Agent and gateway deployment

OpenTelemetry supports different Collector deployment patterns.

A common enterprise approach is to run lightweight Collectors close to applications and forward telemetry to centralized gateway Collectors.

This provides a separation between local telemetry collection and centralized processing.

However, it introduces operational complexity and additional infrastructure. The Collector itself becomes another production component that needs capacity planning, monitoring, and failure handling.

Reference: OpenTelemetry Collector · Gateway Deployment Pattern

11. Sampling: How Do We Control Telemetry Volume?

Agent-based workflows can generate a significant amount of telemetry.

Consider an agent that performs 15 model calls and 20 tool calls during a single task.

If the application handles thousands of tasks per hour, storing every span indefinitely may become expensive.

Sampling helps control this volume.

There are two common approaches.

Head-Based Sampling

The sampling decision is made when the trace starts.

For example, we may decide to sample 10% of incoming requests.

This is relatively simple and reduces the amount of telemetry generated or exported.

However, we do not yet know whether a trace will eventually fail or become unusually slow.

Tail-Based Sampling

The sampling decision is made after considering the trace’s collected spans.

This makes it possible to retain:

• All failed traces.
• All traces exceeding a latency threshold.
• Traces containing specific tool failures.
• A percentage of successful traces.

For AI agents, tail-based sampling can be particularly useful because many execution problems only become visible after several operations have completed.

However, tail sampling has an important infrastructure requirement: all spans belonging to the same trace must reach the same tail-sampling Collector instance.

This generally requires trace-ID-aware routing when using multiple Collector instances.

A practical approach is to combine sampling strategies:

• Preserve error and high-latency traces.
• Retain a representative sample of successful traces.
• Maintain complete aggregate metrics where possible.
• Apply stricter retention policies to verbose telemetry.

Reference: OpenTelemetry Sampling · Tail Sampling Processor

12. Security and Privacy Considerations

This is one of the most important aspects of AI observability in enterprise environments.

Traditional telemetry usually contains operational information such as request duration, HTTP status codes, and service names.

AI telemetry can contain much more sensitive information.

For example:

• User prompts
• System instructions
• Generated responses
• Tool arguments
• Tool results
• Retrieved documents
• Conversation context
• Customer or business identifiers

Sending all this information to an observability backend can create additional privacy and data governance risks.

Do not capture prompts and responses by default

A particularly important detail is that the AI SDK OpenTelemetry integration records inputs and outputs by default unless configured otherwise.

We can disable this behavior:

registerTelemetry(
  new OpenTelemetry({
    recordInputs: false,
    recordOutputs: false,
  }),
);

For many production applications, this is a better starting point.

We can still collect useful operational telemetry:

• Model identifiers
• Token counts
• Duration
• Finish reasons
• Tool execution times
• Error information
• Workflow identifiers

without exporting the complete conversation.

Apply data minimization

Before exporting telemetry, ask whether each attribute is genuinely needed.

Avoid sending:

• Access tokens
• Authorization headers
• Passwords
• Raw customer records
• Full database query results
• Unfiltered tool arguments

For sensitive workflows, use approved metadata rather than full payloads.

Separate operational telemetry from application data

An observability backend should not automatically become another storage system for business data.

The fact that a trace is useful for debugging does not mean the full underlying request and response should be retained indefinitely.

Define access and retention policies

Enterprise environments should define:

• Who can access traces.
• Which teams can view prompt and response content.
• How long telemetry is retained.
• How data is deleted.
• Which environments are allowed to capture full content.
• How telemetry exports are authenticated and encrypted.

A sensible approach is to use more detailed content capture in controlled development environments and much stricter policies in production.

Reference: OpenTelemetry GenAI Content Capture

13. Limitations and Trade-offs

OpenTelemetry provides a strong foundation, but it is not a complete AI observability solution by itself.

Instrumentation coverage varies

Not every framework, provider, or agent runtime exposes the same information.

Some integrations can automatically trace model calls and tool execution, while others require custom instrumentation.

We should verify what is actually captured rather than assuming that installing an OpenTelemetry package provides complete visibility.

GenAI semantic conventions are evolving

The GenAI conventions are still under development.

Attribute names, operation models, and implementation support may evolve.

Teams should pin versions, review changes, and avoid coupling application business logic to conventions that have not reached the required maturity.

Observability does not guarantee answer quality

A trace can explain what happened but does not prove that the model’s result is correct.

Evaluation datasets, deterministic checks, and human feedback remain necessary.

Telemetry has its own cost

Collecting every input, output, tool argument, and execution event can generate significant storage and processing overhead.

We should measure the cost of observability itself.

High-cardinality dimensions can become expensive

Adding unique user IDs, conversation IDs, trace IDs, or request IDs to metric labels can create an excessive number of time series.

Use traces for request-level correlation and keep metric dimensions bounded.

Distributed tracing requires correct context propagation

A2A communication, queues, background tasks, and remote tool execution may cross process boundaries.

If context is lost, we can end up with disconnected traces.

AI evaluation requires careful design

LLM-as-a-Judge can provide useful signals, but it can also be inconsistent or biased.

A single evaluation score should not become the only definition of quality.

Combine deterministic checks, representative datasets, human review, and calibrated evaluators.

14. What Can an AI Chapter Lead Contribute?

I think this is where observability moves from an implementation detail to an engineering capability.

When multiple teams develop AI-powered applications, each team may implement different logging, tracing, and evaluation mechanisms.

Over time, this creates fragmented visibility.

One team measures token usage, another measures response latency, and another stores tool-call logs without a common correlation model.

As an AI Chapter Lead, I would focus on establishing a shared observability standard that can be reused across teams and applications.

1. Define a Common Telemetry Standard

Establish a baseline for the telemetry every agent-based application should produce.

For example:

CategoryRequired informationApplicationService name, environment, versionWorkflowWorkflow name, workflow versionAgentAgent name, execution statusModelProvider, requested model, response modelUsageInput/output tokens, cache usageToolsTool name, duration, execution statusExecutionTrace ID, span ID, error informationBusiness outcomeCompleted, failed, canceled, escalated

The objective is to make observability consistent without forcing every team to use the same agent framework.

2. Build a Shared Instrumentation Package

We can create an internal TypeScript package that wraps the common OpenTelemetry configuration.

For example:

@company/ai-observability
|
+-- OpenTelemetry initialization
+-- AI SDK integration
+-- Agent span utilities
+-- Workflow instrumentation
+-- Tool execution tracking
+-- Cost attribution
+-- Evaluation metadata
+-- Privacy and redaction policies

This package can provide standard helpers for application-level tracing and reduce duplicated implementation.

For teams using .NET, Python, or Go, equivalent instrumentation conventions can be established without requiring identical SDK implementations.

3. Standardize Agent and Workflow Metrics

We should define a common set of metrics that helps teams understand both technical performance and business outcomes.

Examples:

• Agent success rate
• Workflow completion rate
• Tool-call failure rate
• Model latency
• Cost per successful task
• Average token usage
• Human intervention rate
• Evaluation quality scores

A shared dashboard can make it easier to compare systems and identify performance regressions.

4. Establish an Internal Evaluation Framework

Observability without evaluation gives us only part of the picture.

We can create reusable evaluation datasets and testing patterns for common agent behaviors.

For example:

• Tool selection accuracy
• Missing-source detection
• Correct handling of incomplete input
• Permission violations
• Unsupported claims
• Failure recovery
• Task completion quality

This is particularly valuable because agent behavior can change when prompts, tools, models, or orchestration logic are modified.

5. Establish Privacy and Data Governance

Define what can and cannot be captured by default.

The shared implementation should make safe defaults easy to use and prevent teams from accidentally exporting sensitive prompts, responses, or tool payloads.

This should include clear ownership, access policies, retention, and audit requirements.

6. Run a Cross-Team Pilot

A useful pilot would be a multi-agent workflow that involves several independently deployed services.

For example, a merge request review involving coordinator, security, and testing agents.

The pilot should demonstrate:

1. One trace across the complete request.
2. Parent-child relationships between agents.
3. Model and tool execution spans.
4. Token and cost attribution.
5. Error and retry visibility.
6. Evaluation results associated with the completed workflow.

This would provide a reference architecture that other teams can reuse.

7. Make Observability Part of the Development Lifecycle

Observability should not begin after production deployment.

We can introduce telemetry and evaluation requirements into the development process itself.

For example:

• New agents must have execution tracing.
• New tools must expose execution status and duration.
• Prompt changes must pass relevant regression evaluations.
• New workflows must define success and failure criteria.
• Production deployments must have basic operational dashboards.

This makes observability a standard engineering practice rather than a separate monitoring task.

15. A Practical Adoption Roadmap

I would avoid introducing every observability component at once.

A phased approach is more useful.

PhaseFocusExpected result1OpenTelemetry initializationBasic traces and metrics2AI SDK instrumentationModel and tool visibility3Custom agent spansComplete workflow tracing4Cost attributionUsage and cost per task5EvaluationOffline and online quality checks6Security and governancePrivacy and access controls7Shared dashboardsCross-team operational visibility

The first milestone should be simple:

For every important agent workflow, we should be able to answer what happened, how long it took, how much it cost, and whether it achieved the intended result.

Once this foundation is established, more advanced capabilities such as anomaly detection, adaptive sampling, automated evaluation, and cost-aware model routing become much easier to introduce.

Conclusion

AI agents introduce a new level of complexity to application observability.

A single user request can trigger multiple model calls, tool executions, retrieval operations, and remote agent interactions. Monitoring only the HTTP request or final model response is not enough to understand the complete execution.

OpenTelemetry provides a vendor-neutral foundation for collecting traces, metrics, and logs across these components.

Combined with GenAI semantic conventions, AI SDK instrumentation, distributed tracing, and dedicated evaluation systems, it gives engineering teams a much better understanding of how agent-based applications behave in production.

But the most important distinction remains:

Observability tells us what happened. Evaluation tells us whether it was good. Cost attribution tells us whether it was worth it.

For an AI Chapter Lead, the opportunity is to make these capabilities part of the engineering standard: shared instrumentation, consistent metrics, measurable quality, and secure telemetry practices.

The goal should not be to collect more telemetry.

It should be to make every production agent workflow understandable, measurable, and continuously improvable.

References and Further Reading

OpenTelemetry

1. OpenTelemetry Official Documentation — Core concepts, signals, instrumentation, and architecture.
2. OpenTelemetry JavaScript — TypeScript and Node.js instrumentation.
3. OpenTelemetry Tracing — Traces, spans, and distributed execution.
4. OpenTelemetry Context Propagation — Distributed trace context.
5. OpenTelemetry Collector — Receiving, processing, and exporting telemetry.
6. OpenTelemetry Gateway Deployment — Centralized telemetry collection.
7. OpenTelemetry Sampling — Head-based and tail-based sampling.

GenAI Semantic Conventions

1. GenAI Semantic Conventions — Spans — Model operations, retrieval, and tool execution.
2. GenAI Agent Spans — Agent invocation, workflows, and tool spans.
3. GenAI Metrics — Agent duration, tool calls, and workflow metrics.
4. GenAI Token Metrics — Token consumption and usage measurements.

Vercel AI SDK

1. AI SDK Telemetry — OpenTelemetry integration, span hierarchy, and custom attributes.
2. AI SDK v7 Migration Guide — Version-specific telemetry changes.
3. Next.js OpenTelemetry — Application-level instrumentation.

AI Evaluation and Observability Platforms

1. Langfuse Evaluation — Online and offline evaluation.
2. Langfuse Core Concepts — Datasets, experiments, and evaluators.
3. Arize Phoenix — Open-source AI observability and evaluation.
4. OpenInference — OpenTelemetry-based instrumentation conventions for AI applications.

Most requests in a distributed system finish quickly.

A few don’t.

Your API might have a median response time of 40ms, while its p99 latency reaches 900ms. Most users get a fast response, but a small percentage experience significantly longer waits.

These slow requests are particularly problematic in systems that depend on multiple services.

A single slow dependency can delay an entire operation, even when every other component is responding normally.

One way to address this problem is request hedging.

Instead of waiting indefinitely for an unusually slow request, the client starts another copy after a configured delay. Whichever request succeeds first wins, and the remaining requests are canceled.

This technique can significantly reduce tail latency, but it comes with an important trade-off: it generates additional traffic.

If every slow request triggers another request without limits, the mechanism designed to improve latency can overload the dependency it’s trying to protect.

In this article, we’ll explore:

• What tail latency is and why averages hide it
• How hedged requests work
• The difference between hedging and retries
• How to implement hedged requests in Go
• How to cancel losing requests safely
• How to integrate hedging with retry budgets and adaptive concurrency
• When hedging helps and when it makes things worse
• What to monitor and test in production

1. The Problem: Tail Latency

Let’s say an API receives 10,000 requests.

Its response-time distribution looks like this:

PercentileResponse timep5040msp9075msp95120msp99900msp99.92.5s

Looking only at the average or median would suggest that the service is performing well.

But 1% of 10,000 requests means 100 requests take at least 900ms.

For latency-sensitive applications, these outliers matter.

Consider an API that aggregates data from five independent services:

                  API Request
                      |
          +-----------+-----------+
          |           |           |
       Service A  Service B   Service C
          |           |           |
       Service D  Service E       |
          |           |           |
          +-----------+-----------+
                      |
                 Final Response

If the API needs all five responses, its total latency is constrained by the slowest dependency.

Even if four services respond in 30ms, one service taking 800ms can delay the entire operation.

This is one reason tail latency becomes increasingly important as distributed systems grow.

Jeff Dean and Luiz André Barroso discuss this problem in their influential paper, The Tail at Scale. They describe techniques for building responsive services despite unpredictable latency in individual components.

Reference: https://research.google/pubs/the-tail-at-scale/

2. What Are Hedged Requests?

A hedged request is a duplicate request sent while the original request is still in progress.

The client does not immediately send multiple copies.

Instead, it waits for a configurable period.

If the original request hasn’t completed, the client sends another request, ideally to a different replica.

The first successful response wins.

Time
 |
 |  Primary request
 |  ────────────────────────────────>
 |
 |  Wait: 100ms
 |
 |  Hedge request
 |  ─────────────────────>
 |
 |  Hedge response: 160ms
 |  Primary response: 450ms
 |
 |  Return hedge response
 |  Cancel primary request
 |

The important detail is that the second request starts before the first one fails.

This is fundamentally different from a traditional retry.

A retry waits for an unsuccessful attempt before trying again.

A hedge assumes that an unusually slow request might be an outlier and starts another attempt proactively.

Why does this work?

Imagine two replicas:

• Replica A is experiencing temporary contention.
• Replica B is healthy.

The primary request goes to A.

After 100ms, it still hasn’t completed, so the client sends a hedge to B.

If B responds in 60ms, the client returns that response instead of waiting for A.

The total latency becomes approximately 160ms instead of 450ms.

The original request might still finish later, but its result is no longer needed.

3. Hedging vs. Retrying

These mechanisms are related, but they solve slightly different problems.

MechanismWhen the next attempt startsPrimary goalRetryAfter an attempt failsRecover from transient failuresHedgingWhile an attempt is still runningReduce tail latencyTimeoutAfter a deadline expiresBound waiting timeCircuit breakerWhen a dependency is unhealthyStop sending harmful traffic

A retry responds to failure.

A hedge responds to excessive waiting.

This distinction matters because hedging can increase downstream load even when the original request would eventually succeed.

For that reason, hedging should generally be limited to safe, idempotent operations, especially reads.

4. Choosing the Hedging Delay

The hedging delay is one of the most important configuration decisions.

If the delay is too short, the client duplicates too many requests.

If it is too long, the hedge provides little latency improvement.

A reasonable starting point is to use a measured latency percentile for a particular operation.

For example:

PercentileLatencyp5040msp9070msp95100msp99450ms

A 100ms hedging delay means that requests exceeding the p95 threshold become candidates for hedging.

Under stable conditions, approximately 5% of requests may reach that threshold.

This does not mean exactly 5% of requests will generate a hedge: requests can complete at the threshold boundary, and actual behavior depends on the latency distribution and scheduling.

The important principle is to trigger hedging in the tail, not in the middle of normal traffic.

The original Tail at Scale paper discusses using a high-percentile latency threshold to limit duplicate traffic while reducing extreme latency. Its examples demonstrate why carefully selected hedging delays can achieve substantial tail-latency improvements with relatively modest additional load.

Source: https://www.andrew.cmu.edu/user/gaurij/18-847F-Readings/tail_at_scale.pdf

5. Implementing Hedged Requests in Go

Let’s build a small hedging executor.

Our implementation will support:

• An immediate primary request
• A configurable hedging delay
• A maximum number of attempts
• Context cancellation
• First-successful-response semantics
• Retryable error classification
• An optional shared retry budget

We’ll use Go generics so the executor can return any response type.

Step 1: Define the policy

package hedge

import (
    "context"
    "errors"
    "fmt"
    "time"
)

var (
    ErrInvalidPolicy = errors.New("invalid hedging policy")
    ErrBudgetDenied  = errors.New("hedging budget exhausted")
)

type Budget interface {
    TryTake() bool
}

type Policy struct {
    Delay       time.Duration
    MaxAttempts int
    Budget      Budget
    ShouldHedge func(error) bool
}

type result[T any] struct {
    value T
    err   error
}

The Budget interface allows us to reuse the retry budget implementation from the previous article.

The original request does not consume a hedge token.

Every additional request does.

We also use ShouldHedge to distinguish errors that should terminate the operation from errors that may allow another attempt.

For example, a validation error should not trigger another request.

Step 2: Implement the executor

func Do[T any](
    ctx context.Context,
    policy Policy,
    operation func(context.Context, int) (T, error),
) (T, error) {
    var zero T

    if policy.Delay <= 0 ||
        policy.MaxAttempts < 1 ||
        policy.MaxAttempts > 3 ||
        operation == nil ||
        policy.ShouldHedge == nil {
        return zero, ErrInvalidPolicy
    }

    if err := ctx.Err(); err != nil {
        return zero, err
    }

    hedgeCtx, cancel := context.WithCancel(ctx)

    defer cancel()

    results := make(chan result[T], policy.MaxAttempts)
    launched := 0
    inFlight := 0

    var lastErr error

    startAttempt := func() {
        launched++
        inFlight++
        attempt := launched
        go func() {
            value, err := operation(hedgeCtx, attempt)
            results <- result[T]{
                value: value,
                err:   err,
            }
        }()
    }

    // The primary request starts immediately.
    startAttempt()

    var timer *time.Timer

    var timerC <-chan time.Time

    if policy.MaxAttempts > 1 {
        timer = time.NewTimer(policy.Delay)
        timerC = timer.C
        defer timer.Stop()
    }

    for {
        select {
        case <-ctx.Done():
            return zero, ctx.Err()
        case res := <-results:
            inFlight--
            if res.err == nil {
                // First successful response wins.
                return res.value, nil
            }

            lastErr = res.err

            if !policy.ShouldHedge(res.err) {
                return zero, res.err
            }

            if inFlight == 0 &&
                launched == policy.MaxAttempts {
                return zero, lastErr
            }
        case <-timerC:
            if launched >= policy.MaxAttempts {
                timerC = nil
                continue
            }

            if policy.Budget != nil &&
                !policy.Budget.TryTake() {
                // No more additional requests.
                timerC = nil
                if inFlight == 0 {
                    if lastErr != nil {
                        return zero, lastErr
                    }
                    return zero, ErrBudgetDenied
                }
                continue
            }

            startAttempt()

            if launched < policy.MaxAttempts {
                timer.Reset(policy.Delay)
            } else {
                timerC = nil
            }
        }
    }
}

Let’s examine the important decisions.

The primary request starts immediately.

We don’t wait for a timer before sending the first request.

Additional requests start only after the configured delay.

If the primary finishes successfully before the delay expires, the executor returns without sending a hedge.

The first successful response wins.

As soon as a successful result arrives, the function returns and cancels the shared context.

Errors are classified.

A fatal error can terminate the operation immediately. A retryable error can be treated as non-final while another attempt is still running.

The number of attempts is bounded.

The implementation intentionally limits the example to three total attempts, including the original request.

This is a conservative default for the example, not a universal limit.

Step 3: Use the executor

Imagine a service that reads market data from multiple replicas.

type Price struct {
    Symbol string
    Value  float64
}

func fetchPrice(
    ctx context.Context,
    attempt int,
) (Price, error) {
    replica := selectReplica(attempt)
    return replica.Fetch(ctx, "AAPL")
}

We can now use the hedging executor:

policy := hedge.Policy{
    Delay:       100 * time.Millisecond,
    MaxAttempts: 2,
    Budget:      retryBudget,
    ShouldHedge: isRetryableError,
}

price, err := hedge.Do(
    ctx,
    policy,
    fetchPrice,
)

if err != nil {
    return err
}

fmt.Printf(
    "%s: %.2f\n",
    price.Symbol,
    price.Value,
)

The primary request goes to the first replica.

If it hasn’t completed after 100ms, the executor checks the budget and starts another request.

If the second replica responds first, the result is returned and the first request is canceled.

The example assumes that selectReplica routes attempts to distinct, suitable replicas. Sending both copies to the same overloaded instance may provide little benefit.

6. Cancellation Is Not the Same as Stopping Work

This is one of the most important details in a hedging implementation.

When the winning request completes, the executor calls cancel() on the shared context.

The losing requests receive cancellation through their contexts.

But cancellation is cooperative.

It does not forcibly terminate a goroutine, an HTTP request, a database query, or server-side processing.

For example, a dependency may continue executing a query even after the client disconnects.

A correct operation should propagate its context:

func fetch(
    ctx context.Context,
    client *http.Client,
) error {
    req, err := http.NewRequestWithContext(
        ctx,
        http.MethodGet,
        "https://api.example.com/prices/AAPL",
        nil,
    )
    if err != nil {
        return err
    }

    resp, err := client.Do(req)

    if err != nil {
        return err
    }

    defer resp.Body.Close()

    return nil
}

The server should also respect request cancellation where possible.

Otherwise, the client may stop waiting while the dependency continues doing unnecessary work.

This is especially important for expensive database queries and CPU-intensive operations.

7. Why a Hedge Budget Matters

Hedging creates additional traffic.

Imagine a service receiving 10,000 requests per second.

If 5% reach the hedging threshold, that could mean approximately 500 additional attempts per second.

Now imagine the dependency is already degraded.

More concurrent requests may increase queueing, which increases latency, which triggers more hedges.

The mechanism can create the same kind of amplification problem that we discussed in the Retry Budgets article.

This is why the hedge budget should be shared across requests.

A simple rule:

• The original request is always allowed, subject to normal admission controls.
• Each additional hedge requires a token.
• Successful normal traffic replenishes the budget.
• If the budget is exhausted, no new hedges are launched.

For example:

budget := retry.NewRetryBudget(
    20,
    0.1,
)

The exact budget should be tuned according to the dependency’s spare capacity and the service’s traffic patterns.

The important part is that hedging is not an unlimited permission to duplicate requests.

It is an additional workload that must be controlled.

8. Hedging and Adaptive Concurrency

Hedging and adaptive concurrency work particularly well together, but they need to share a consistent view of downstream capacity.

Adaptive concurrency adjusts how many requests are allowed in flight based on observed dependency behavior.

Hedging creates additional in-flight requests.

If the adaptive limiter counts only original requests, hedged requests can bypass the intended concurrency ceiling.

That defeats the purpose of the limiter.

A safer integration is:

1. The primary request acquires a concurrency permit.
2. A hedge must acquire its own permit.
3. If no permit is available, the hedge is skipped.
4. The winning request returns its result.
5. Every attempt releases its own permit when it actually completes.

There is an important implementation detail here.

Do not release the losing request’s concurrency permit merely because the caller stopped waiting for it.

The permit must remain held until the underlying operation actually completes or otherwise confirms that the protected work has stopped.

Otherwise, the limiter undercounts active downstream work.

This is also why a shared cancellation context alone is not enough to guarantee that hedged work has disappeared.

9. Hedging Requires Idempotency

Sending the same operation more than once is not always safe.

Consider these operations:

OperationHedging suitabilityRead a market priceGenerally suitableRead a portfolioGenerally suitableFetch cached configurationGenerally suitableCreate a paymentUnsafe without idempotencySubmit an orderUnsafe without deduplicationSend an emailMay produce duplicatesUpdate an account balanceUnsafe without strong safeguards

A hedge can reach two different servers before either has completed.

Both servers may execute the operation.

Canceling one request does not guarantee that its side effects are rolled back.

For operations with side effects, use an idempotency key or another server-side deduplication mechanism.

Even then, the server must correctly enforce the idempotency contract.

Hedging should generally be introduced first for read-only operations where duplicate execution is safe.

10. Where Should Hedged Requests Be Sent?

Sending a duplicate request to the same server is often ineffective.

If that server is experiencing CPU contention, a long queue, or an overloaded database connection pool, the second request may suffer from the same problem.

Ideally, the hedge should target an independent replica.

But independence is not guaranteed.

Two replicas may share:

• The same database
• The same network path
• The same cache
• The same storage system
• The same overloaded region

This is called correlated failure or correlated latency.

If both requests depend on the same struggling resource, hedging may increase load without meaningfully reducing latency.

Replica selection should consider failure domains and actual dependency topology.

11. Observability: Measure the Cost of Hedging

A hedging system needs more than a request-latency dashboard.

You need to understand whether the reduction in tail latency justifies the additional work.

Useful metrics include:

MetricTypePurposehedged_requests_totalCounterNumber of additional attemptshedge_wins_totalCounterRequests where a hedge returned firsthedge_budget_denied_totalCounterHedges rejected by the budgethedge_canceled_totalCounterLosing attempts canceled by the clienthedge_attempts_totalCounterTotal attempts, including originalshedge_delay_secondsHistogramTime before a hedge is launchedrequest_latency_secondsHistogramEnd-to-end request latency

A useful derived metric is the hedge win rate:

[ HedgeWinRate = \frac{HedgeWins}{HedgedRequests} ]

If hedges rarely win, you may be generating additional traffic without much benefit.

Also compare:

• p50 latency
• p95 latency
• p99 latency
• Downstream request volume
• Dependency CPU and memory usage
• Cancellation effectiveness
• Error rate

The goal is not to minimize p99 at any cost.

It is to improve tail latency while keeping the additional workload within the dependency’s capacity.

12. Testing Hedged Requests

Concurrency behavior needs deterministic tests.

At minimum, test these scenarios:

• The primary request succeeds before the hedging delay.
• The primary request is slow and the hedge wins.
• The primary request fails with a retryable error.
• A fatal error cancels outstanding attempts.
• The hedge budget prevents additional requests.
• The parent context is canceled.
• The maximum number of attempts is respected.
• Losing operations receive cancellation.
• A losing operation that ignores cancellation does not release its concurrency permit prematurely.

A simple test for the winning hedge could use controlled channels instead of arbitrary sleeps:

func TestHedgeWins(t *testing.T) {
    primaryStarted := make(chan struct{})
    hedgeStarted := make(chan struct{})

    ctx, cancel := context.WithTimeout(
        context.Background(),
        time.Second,
    )

    defer cancel()

    policy := Policy{
        Delay:       20 * time.Millisecond,
        MaxAttempts: 2,
        ShouldHedge: func(error) bool {
            return true
        },
    }

    result, err := Do(
        ctx,
        policy,
        func(
            ctx context.Context,
            attempt int,
        ) (string, error) {
            if attempt == 1 {
                close(primaryStarted)
                <-ctx.Done()
                return "", ctx.Err()
            }
            close(hedgeStarted)
            return "hedge response", nil
        },
    )

    if err != nil {
        t.Fatal(err)
    }

    if result != "hedge response" {
        t.Fatalf(
            "unexpected result: %s",
            result,
        )
    }

    <-primaryStarted
    <-hedgeStarted
}

The test verifies that the second request can win and that the operation completes successfully.

In a complete test suite, also verify that the primary operation observes cancellation.

Run the race detector:

go test -race ./...

Finally, use load tests with a dependency that deliberately introduces variable latency.

A useful experiment compares:

1. Hedging disabled.
2. Hedging enabled with a conservative delay.
3. Hedging enabled with an aggressive delay.
4. Hedging enabled with a shared budget.
5. Hedging enabled with adaptive concurrency.

Compare both latency improvements and additional downstream traffic.

13. Production Checklist

Before introducing hedged requests into a production service:

• [ ] Hedge only safe, idempotent operations.
• [ ] Select a delay from measured latency distributions.
• [ ] Use a strict maximum attempt count.
• [ ] Require a shared budget for additional attempts.
• [ ] Ensure hedges target independent replicas where possible.
• [ ] Apply concurrency limits to every attempt.
• [ ] Propagate context cancellation.
• [ ] Ensure permits remain held while work is still active.
• [ ] Respect the original request deadline.
• [ ] Measure hedge win rate and additional traffic.
• [ ] Test correlated latency and dependency overload.
• [ ] Roll out gradually and compare against a no-hedging baseline.

Conclusion

Tail latency is one of the most difficult problems in distributed systems because a small number of unusually slow operations can dominate the response time of an entire request.

Hedged requests provide a way to reduce that impact.

By launching a second request after a carefully selected delay, a client can avoid waiting for an unusually slow replica and use the first successful response.

But hedging is not free.

Every additional request consumes capacity, and uncontrolled duplication can make an already degraded dependency even slower.

The right approach combines measured hedging delays, idempotent operations, cancellation, retry budgets, and adaptive concurrency limits.

The goal isn’t to send more requests. It’s to get the response sooner without making the system slower for everyone else.

References

• The Tail at Scale — Google Research
• Request Hedging — gRPC Documentation
• gRPC Retry and Hedging Design

Hedged Requests in Go: Reducing Tail Latency Without Overloading Dependencies was originally published in Alt. on Medium, where people are continuing the conversation by highlighting and responding to this story.

How event-driven agent interactions and generative UI are changing the way we build AI-powered frontend applications.

1. Why Are Chat Interfaces No Longer Enough?

Most AI applications start with a familiar interface: a text input, a message history, and a streaming response.

This works well for question answering, summarization, and conversational experiences. However, agent-based applications introduce a different set of interaction requirements.

Consider an AI assistant that analyzes a user’s investment portfolio, investigates market movements, or prepares a technical report.

A conventional chat interface might return a response containing:

• A summary of the analysis
• A list of recommendations
• Several data points
• A suggested next action

But what if the user needs more than a textual response?

What if the application needs to display:

• Interactive charts
• Dynamic forms
• Data tables
• Approval workflows
• Progress indicators
• Context-aware actions
• Multi-step interfaces that evolve as the agent works

We could build every possible interface in advance and ask the model to select the appropriate components. This approach works for predictable workflows, but it becomes restrictive when the interaction itself is dynamic.

Another option is to allow the model to generate HTML, React components, or JavaScript directly. However, executing arbitrary model-generated code introduces significant security, maintainability, and governance challenges.

This is where two interesting technologies enter the picture:

AG-UI (Agent-User Interaction Protocol) and A2UI (Agent-to-User Interface).

Although their names are similar, they address different problems.

AG-UI standardizes how agents communicate with user-facing applications. A2UI provides a declarative format for describing interfaces that client applications can safely render using their own component libraries.

Together, they provide an interesting foundation for building agent-powered applications beyond traditional chat interfaces.

References: AG-UI Documentation · Google A2UI

2. What Is AG-UI?

AG-UI stands for Agent-User Interaction Protocol.

It is an open, lightweight, event-based protocol designed to standardize communication between AI agents and frontend applications.

The underlying problem is straightforward.

An agent does not always produce one response and finish. It may execute multiple tools, delegate work, update its state, request additional input, and continue processing.

A traditional request-response model does not necessarily represent these interactions well.

For example, imagine an agent performing a database analysis:

1. The user requests an analysis.
2. The agent starts executing the task.
3. It calls a database tool.
4. The query completes.
5. The agent processes the returned data.
6. It generates a chart.
7. It asks the user to confirm an additional filter.
8. It completes the analysis.

If the frontend only receives the final response, the user has no visibility into what happened during execution.

We could implement custom WebSocket messages, HTTP streaming events, and application-specific state synchronization. But every application would need to define and maintain its own communication contract.

AG-UI provides a standardized event model for this interaction.

It allows agent backends to communicate execution events, messages, tool calls, state changes, and activity updates to the frontend.

The protocol is intentionally not tied to one agent framework or frontend technology.

You can build the backend using LangGraph, Google ADK, CrewAI, or a custom runtime, while the frontend uses React or another supported client implementation.

The important point is that AG-UI does not define how an agent reasons or executes its internal logic. It defines how that execution is communicated to the user-facing application.

Reference: AG-UI — Introduction

3. How Does AG-UI Work?

AG-UI follows an event-driven architecture.

Instead of treating an agent interaction as a single response, the backend emits a sequence of typed events that the frontend consumes.

A simplified architecture looks like this:

                  User
                   |
                   v
             React Application
                   |
                   v
               AG-UI Client
                   |
            Event Stream / SSE
                   |
                   v
              Agent Runtime
                   |
          +--------+--------+
          |        |        |
          v        v        v
        LLM      Tools    Subagents
          |        |        |
          +--------+--------+
                   |
                   v
             Agent Events
                   |
                   v
              AG-UI Client
                   |
                   v
              Update UI

The frontend and backend have separate responsibilities.

Backend responsibilities:

• Execute agent logic.
• Handle model interactions.
• Execute tools.
• Maintain execution state.
• Emit standardized events.
• Handle additional user input and resume interactions.

Frontend responsibilities:

• Initiate agent runs.
• Consume event streams.
• Render messages and activity updates.
• Synchronize application state.
• Present tool activity.
• Handle user interactions and approvals.

AG-UI is transport-agnostic. Common integration patterns include Server-Sent Events (SSE), WebSockets, and webhooks.

The official HttpAgent client supports HTTP SSE and an HTTP binary transport.

This separation means we do not have to make frontend components understand the implementation details of the backend agent framework.

Reference: AG-UI — Core Architecture

4. Understanding AG-UI Events

Events are the foundation of AG-UI.

Rather than sending an unstructured text response, an agent can communicate different types of events depending on what is happening during execution.

The main event categories include:

4.1. Lifecycle Events

These events describe the execution lifecycle.

EventPurposeRUN_STARTEDAn agent run has startedRUN_FINISHEDAn agent run has finishedRUN_ERRORAn execution error occurredSTEP_STARTEDA new execution step startedSTEP_FINISHEDAn execution step finished

This allows the frontend to display meaningful execution states instead of relying on a generic loading indicator.

4.2. Text Message Events

These events represent streaming messages.

EventPurposeTEXT_MESSAGE_STARTStarts a messageTEXT_MESSAGE_CONTENTSends a text chunkTEXT_MESSAGE_ENDCompletes a message

The frontend can append each text chunk as it arrives.

This provides a familiar streaming experience while keeping the underlying event structure explicit.

4.3. Tool Call Events

Tool execution is an important part of agent workflows.

AG-UI defines events such as:

• TOOL_CALL_START
• TOOL_CALL_ARGS
• TOOL_CALL_END
• TOOL_CALL_RESULT

These events allow the frontend to display which tool is being used and follow its execution lifecycle.

For example, an interface might display:

Analyzing repository...
    |
    v
GitLab: Fetching merge request
    |
    v
Code Review: Analyzing changes
    |
    v
Security Check: Completed
    |
    v
Generating report

One distinction is important: receiving a tool-call event does not automatically mean the frontend should execute that tool. Tool execution and authorization remain responsibilities of the appropriate runtime or application layer.

4.4. State Management Events

AG-UI also defines events for synchronizing application and agent state.

EventPurposeSTATE_SNAPSHOTSends a complete state snapshotSTATE_DELTASends incremental state changesMESSAGES_SNAPSHOTSynchronizes conversation history

For example, an agent might update the current analysis status:

{
  "type": "STATE_SNAPSHOT",
  "snapshot": {
    "status": "analyzing",
    "completedSteps": 3,
    "totalSteps": 5,
    "currentTask": "Evaluating transaction data"
  }
}

The frontend can consume this state and update progress indicators, dashboards, or other components.

4.5. Activity and Subagent Events

Modern agent systems often involve multiple agents working together.

AG-UI includes activity and subagent-related events, allowing a frontend to represent more complex execution patterns.

For example:

• A research agent starts working.
• A data analysis agent receives a delegated task.
• The security agent completes its review.
• The coordinator produces the final result.

This makes it possible to build interfaces that expose useful execution activity without requiring users to understand the internal implementation of each agent.

The current AG-UI v1.0 schema documents event families for lifecycle, text, tools, state, activity, subagents, reasoning, and custom events.

A critical implementation consideration: reasoning-related events should not be treated as permission to expose raw model chain-of-thought. User-facing applications should expose approved progress summaries and operational status instead.

Reference: AG-UI — Event Types and Specification

5. How to Integrate AG-UI with a React Application

AG-UI provides a TypeScript SDK that can be used to connect frontend applications to compatible agent endpoints.

The official package is @ag-ui/client.

Step 1: Install the package

pnpm add @ag-ui/client @ag-ui/core

Step 2: Create an HTTP agent

The HttpAgent class provides a client for communicating with an AG-UI-compatible backend.

import { HttpAgent } from '@ag-ui/client';

const agent = new HttpAgent({
  url: '/api/agent',
  agentId: 'portfolio-assistant',
  threadId: crypto.randomUUID(),
});

The configuration defines the backend endpoint and identifies the agent and conversation thread.

Step 3: Send a message and consume events

We can use the agent’s addMessage and runAgent methods to start an interaction.

import { HttpAgent } from '@ag-ui/client';
import { EventType } from '@ag-ui/core';

const agent = new HttpAgent({
  url: '/api/agent',
  agentId: 'portfolio-assistant',
  threadId: crypto.randomUUID(),
});

agent.addMessage({
  id: crypto.randomUUID(),
  role: 'user',
  content: 'Analyze my portfolio and summarize its performance.',
});

await agent.runAgent(
  {
    tools: [],
    context: [],
  },
  {
    onEvent: ({ event }) => {
      switch (event.type) {
        case EventType.RUN_STARTED:
          console.log('Agent execution started');
          break;
        case EventType.TEXT_MESSAGE_CONTENT:
          console.log(event.delta);
          break;
        case EventType.TOOL_CALL_START:
          console.log('Tool started:', event.toolCallName);
          break;
        case EventType.RUN_FINISHED:
          console.log('Agent execution completed');
          break;
        case EventType.RUN_ERROR:
          console.error('Agent execution failed');
          break;
      }
    },
  },
);

The example demonstrates the client-side event handling pattern. In a complete application, event handlers would update React state, render tool activity, and maintain conversation state.

The agent SDK also supports shared state, event subscriptions, middleware, and cancellation through abortRun().

Step 4: Synchronize agent state with React

One of the most useful capabilities is the ability to keep frontend and agent state synchronized.

For example, an agent can update task progress while the frontend renders the current status.

With a framework integration such as CopilotKit, this can be implemented through hooks such as useAgent.

Conceptually:

import { useAgent } from '@copilotkit/react-core/v2';

function TaskProgress() {
  const { agent } = useAgent();

  const progress = agent.state.progress ?? {
    completed: 0,
    total: 0,
  };

  return (
    <section>
      <h3>Agent Progress</h3>
      <p>
        {progress.completed} / {progress.total} completed
      </p>
    </section>
  );
}

The component reacts to changes in the agent’s shared state.

The important architectural detail is that AG-UI defines the communication contract, while libraries such as CopilotKit provide additional frontend integration and application-level abstractions.

We are not required to use CopilotKit to implement AG-UI. A custom frontend can consume the protocol directly.

References: AG-UI TypeScript SDK · CopilotKit Shared State

6. What Is A2UI?

AG-UI solves the communication problem, but there is another question:

What if an agent needs to generate an actual user interface instead of returning text?

A2UI stands for Agent-to-User Interface.

It is an open-source project initiated by Google that provides a declarative protocol for generating and updating user interfaces from structured JSON messages.

The central idea is simple.

Instead of asking an LLM to generate HTML, JavaScript, or React code, we let it describe what the interface should contain.

For example, an agent may want to display:

• A title
• A summary card
• A table
• A date picker
• A confirmation button

It can describe these elements using A2UI messages.

The frontend then maps those descriptions to its own approved components.

This creates a separation between UI generation and UI execution.

The agent describes the interface. The application decides how that interface is rendered.

Why is this important?

Imagine allowing an LLM to generate the following:

function Dashboard() {
  // Arbitrary model-generated code
}

Executing model-generated React code directly would create substantial security and maintenance problems.

Instead, we can let the model return structured data:

{
  "component": "Card",
  "children": [
    {
      "component": "Text",
      "text": "Portfolio Summary"
    }
  ]
}

The frontend maps Card and Text to components that already exist in its design system.

The model never needs to execute arbitrary JavaScript to create the interface.

This is the main architectural idea behind A2UI.

Reference: Google A2UI — Official Documentation

7. How Does A2UI Work?

A2UI uses a JSON-based streaming protocol.

The agent sends messages to the frontend, and the frontend processes those messages to create or update UI surfaces.

A2UI v0.9.1 defines four primary message types.

7.1. createSurface

Creates a new UI surface.

A surface represents a UI region that contains components and associated data.

{
  "version": "v0.9.1",
  "createSurface": {
    "surfaceId": "portfolio-dashboard",
    "catalogId": "https://a2ui.org/specification/v0_9_1/catalogs/basic/catalog.json"
  }
}

7.2. updateComponents

Defines or updates the components within a surface.

The components are represented as a flat list with references that define their relationships.

For example:

{
  "version": "v0.9.1",
  "updateComponents": {
    "surfaceId": "portfolio-dashboard",
    "components": [
      {
        "id": "root",
        "component": "Column",
        "children": [
          "title",
          "summary"
        ]
      },
      {
        "id": "title",
        "component": "Text",
        "text": "Portfolio Overview",
        "variant": "h1"
      },
      {
        "id": "summary",
        "component": "Text",
        "text": "Your portfolio analysis is ready."
      }
    ]
  }
}

The renderer understands the component definitions and constructs the corresponding UI tree.

7.3. updateDataModel

Updates the data displayed by the components.

A2UI separates component structure from application data.

This is particularly useful because changing data does not necessarily require sending the entire component tree again.

For example:

{
  "version": "v0.9.1",
  "updateDataModel": {
    "surfaceId": "portfolio-dashboard",
    "path": "/portfolio",
    "value": {
      "totalValue": 125000,
      "currency": "USD",
      "change": 4.2
    }
  }
}

Components can reference values in the data model using JSON Pointer paths.

When the data model changes, the renderer can update the relevant parts of the interface.

7.4. deleteSurface

Removes a previously created surface and its associated components and data.

{
  "version": "v0.9.1",
  "deleteSurface": {
    "surfaceId": "portfolio-dashboard"
  }
}

This is useful for managing dynamic UI regions as an interaction evolves.

The complete flow

User Request
     |
     v
Agent Runtime
     |
     v
LLM Generates A2UI Messages
     |
     v
    JSON Stream
     |
     v
A2UI Message Processor
     |
     v
Component Catalog
     |
     v
React / Flutter / Angular Renderer
     |
     v
Interactive User Interface

The process can happen incrementally.

The agent can create a surface first, add components later, and update the data model as more information becomes available.

This is different from generating an entire interface in one response.

References: A2UI Protocol v0.9.1 · A2UI Quickstart

8. Integrating A2UI with React

A2UI provides an official React renderer and a shared web core library.

The relevant packages are:

• @a2ui/react
• @a2ui/web_core

Step 1: Install the packages

pnpm add @a2ui/react @a2ui/web_core

Step 2: Create a MessageProcessor

The MessageProcessor handles protocol messages, surface lifecycle, component definitions, and data models.

For the v0.9 implementation:

import { useState } from 'react';

import {
  MessageProcessor,
} from '@a2ui/web_core/v0_9';

import {
  A2uiSurface,
  basicCatalog,
} from '@a2ui/react/v0_9';

The processor is initialized with the component catalog that the application allows the agent to use.

const processor = new MessageProcessor([
  basicCatalog,
]);

Step 3: Process agent-generated messages

For example, we can create a surface and populate it with components:

processor.processMessages([
  {
    version: 'v0.9',
    createSurface: {
      surfaceId: 'main',
      catalogId: basicCatalog.id,
    },
  },
  {
    version: 'v0.9',
    updateComponents: {
      surfaceId: 'main',
      components: [
        {
          id: 'root',
          component: 'Column',
          children: ['heading', 'description'],
        },
        {
          id: 'heading',
          component: 'Text',
          text: 'Hello from A2UI',
          variant: 'h1',
        },
        {
          id: 'description',
          component: 'Text',
          text: 'This interface was described by an agent.',
        },
      ],
    },
  },
]);

The MessageProcessor maintains the surface and its component model.

Step 4: Render the surface

Once the surface has been created, the React renderer can display it.

function DynamicUI() {
  const [processor] = useState(() => {
    const instance = new MessageProcessor([
      basicCatalog,
    ]);

    instance.processMessages([
      {
        version: 'v0.9',
        createSurface: {
          surfaceId: 'main',
          catalogId: basicCatalog.id,
        },
      },
      {
        version: 'v0.9',
        updateComponents: {
          surfaceId: 'main',
          components: [
            {
              id: 'root',
              component: 'Column',
              children: ['title'],
            },
            {
              id: 'title',
              component: 'Text',
              text: 'Generated Interface',
              variant: 'h1',
            },
          ],
        },
      },
    ]);

    return instance;
  });

  const surface = processor.model.surfacesMap.get('main');

  if (!surface) {
    return <p>Preparing interface...</p>;
  }

  return <A2uiSurface surface={surface} />;
}

This is a minimal rendering example. A production implementation should create and manage the processor at the appropriate application or session scope, rather than reconstructing it for each update.

As new A2UI messages arrive, the application can pass them to the same processor. The surface model drives the React renderer’s updates.

Step 5: Handle user interactions

An interactive UI needs a return path.

For example, the agent may generate a form containing a confirmation button.

When the user clicks that button, the frontend captures the associated action and sends it back to the agent through the selected transport.

The flow becomes:

Agent Generates UI
       |
       v
A2UI Renderer
       |
       v
User Interacts
       |
       v
Frontend Captures Action
       |
       v
Application / Agent Endpoint
       |
       v
Agent Processes Input
       |
       v
New A2UI Messages
       |
       v
Updated Interface

A2UI v0.9.1 uses an external communication channel for these return interactions. Its v1.0 candidate specification introduces additional structured function-call and response mechanisms.

This distinction matters when designing the frontend-backend contract.

References: A2UI React Renderer · A2UI Client Setup

9. What Is the Difference Between AG-UI and A2UI?

This is probably the most important distinction in the article.

AG-UI and A2UI are complementary, but they are not interchangeable.

AreaAG-UIA2UIPrimary purposeAgent-frontend communicationDeclarative UI generationMain abstractionTyped eventsUI surfaces and componentsData flowRun events and state updatesUI structure and data updatesMain responsibilityInteraction lifecycleInterface representationFrontend roleConsume and handle eventsRender approved componentsUser interactionAgent runs, tools, state and interruptsUI actions and data context

Consider an agent that needs to create a dynamic approval dashboard.

AG-UI can communicate:

• The agent has started processing.
• The data retrieval step is complete.
• A tool has returned results.
• The agent has updated its state.
• A user approval is required.

A2UI can describe:

• An approval summary card.
• A table of requested changes.
• A confirmation button.
• A detailed information panel.

The frontend can render both the interaction lifecycle and the generated interface.

Can we use them together?

Yes.

A possible architecture looks like this:

                 User
                  |
                  v
            React Frontend
                  |
                  v
               AG-UI
             Event Stream
                  |
                  v
             Agent Runtime
                  |
                  v
                 LLM
                  |
            +-----+-----+
            |           |
            v           v
        AG-UI Events   A2UI JSON
            |           |
            +-----+-----+
                  |
                  v
            Frontend Host
                  |
                  v
            A2UI Renderer
                  |
                  v
           Native React UI

AG-UI manages the interaction boundary. A2UI describes the interface that should be rendered.

One protocol does not replace the other.

A2UI can also be transported over A2A, MCP-related integrations, SSE, or WebSockets. It is designed to remain independent of a particular transport.

References: AG-UI Architecture · A2UI Transport Decoupling

10. How Do They Fit Into the Existing AI Ecosystem?

We have already discussed MCP, ACP, and A2A in previous articles.

Adding AG-UI and A2UI gives us another important architectural layer.

TechnologyResponsibilityMCPAgent ↔ Tools and external dataACPCoding Agent ↔ Client ApplicationA2AAgent ↔ AgentAG-UIAgent ↔ Frontend interactionA2UIAgent → Declarative UI

A typical enterprise AI platform could use several of these technologies simultaneously.

For example:

1. An application communicates with an agent using AG-UI.
2. The agent accesses internal services through MCP.
3. The agent delegates a specialized task to another agent using A2A.
4. The agent returns dynamic interface definitions using A2UI.
5. The application renders those definitions through its own component system.

ACP can be used independently when we need to integrate coding agents with a client application.

This is not about adopting every protocol. It is about keeping architectural boundaries clear.

11. How Do These Approaches Compare With Traditional Generative UI?

AG-UI and A2UI are not the only ways to build agent-powered interfaces.

There are several approaches worth understanding.

AI SDK Tool-Based UI

Vercel AI SDK already allows applications to render custom UI based on tool calls and structured message parts.

For example, a model can call a weather tool, and the frontend can render a weather card instead of displaying only raw text.

This is a good approach when:

• The application has a known set of components.
• UI behavior is closely tied to model tool calls.
• The application owns the entire interaction contract.

However, it does not automatically provide the same cross-framework UI specification and agent interaction model as AG-UI and A2UI.

MCP Apps

MCP Apps extend MCP tools with interactive UI experiences.

A tool can expose a ui:// resource containing its interface, which an MCP host can render in a sandboxed iframe.

This is particularly useful when a tool needs to provide its own interactive experience, such as a dashboard, form, or specialized workflow.

It solves a different problem from A2UI: the UI is associated with a tool rather than being a general declarative UI surface generated by an agent.

A2UI

A2UI focuses on agent-generated interface descriptions and framework-specific rendering.

The same declarative description can be consumed by compatible clients with different native component implementations.

AG-UI

AG-UI focuses on the interaction protocol rather than the visual representation.

It can be used with ordinary chat interfaces, custom React components, or declarative generative UI approaches.

ApproachBest suited forAI SDK Tool UIApplication-defined component renderingMCP AppsInteractive UIs provided by individual toolsA2UIAgent-generated, declarative interfacesAG-UIStandardized agent-frontend communication

These approaches are not mutually exclusive.

References: AI SDK — useChat · Vercel — MCP Apps Integration · Google A2UI

12. Advantages and Trade-offs

Introducing these technologies can improve architecture and developer experience, but they also introduce new responsibilities.

AG-UI Advantages

Standardized event communication

Teams can consume a consistent event model instead of maintaining separate custom streaming contracts.

Frontend-backend separation

The frontend does not need to understand the internal implementation of each agent framework.

Better visibility into execution

Tool calls, progress updates, and agent state can be represented in the UI.

Framework interoperability

Different backend agent implementations can expose a common interaction contract.

Support for human-in-the-loop workflows

Applications can build approval, clarification, and user-steering experiences around agent execution.

AG-UI Trade-offs

• Existing agent frameworks may require adapters or event translation.
• Teams must manage protocol versions and event compatibility.
• Application state and agent state can become inconsistent if ownership is unclear.
• Authentication, authorization, and event validation remain application responsibilities.
• For simple chat applications, a custom endpoint or ordinary AI SDK integration may be sufficient.

A2UI Advantages

Declarative and safer UI generation

Agents send structured data rather than executable React or JavaScript code.

Framework independence

The same UI description can be mapped to different client-side component systems.

Reuse of existing design systems

Applications can expose their own approved component catalog.

Incremental rendering

The UI structure and data can be updated without resending everything.

Separation of structure and data

Components can remain stable while the agent updates the underlying data model.

A2UI Trade-offs

• A component catalog must be designed and maintained.
• The renderer must support the required component types.
• Complex application-specific interactions require additional integration.
• Schema and protocol version compatibility require attention.
• Debugging streamed component updates can be more difficult than debugging ordinary React components.
• Cross-platform rendering does not guarantee identical visual behavior on every platform.

An important point: declarative JSON is safer than arbitrary executable code, but it is not automatically trustworthy.

The application must still validate messages, restrict available components, and authorize actions.

13. Security and Production Considerations

This is where the difference between a successful demo and a maintainable production implementation becomes important.

1. Never execute arbitrary generated code

An agent should not be able to send arbitrary JavaScript, HTML, or React code and have the application execute it.

A2UI’s declarative component catalog provides a safer approach.

The application should only render components it explicitly supports.

2. Validate incoming messages

AG-UI events and A2UI messages must be treated as untrusted input.

Validate event structures, protocol versions, component definitions, data paths, and payload sizes before processing them.

3. Control available components

Not every component should be available to an agent.

For example, an internal banking application might allow:

• Summary cards
• Data tables
• Charts
• Read-only forms
• Approval request components

However, a sensitive transaction operation should not become available merely because the agent can generate a button.

The backend must authorize the actual business operation.

4. Keep application state under application control

There should be a clear distinction between UI state, agent state, and authoritative business state.

An agent-generated UI should not be considered a source of truth for financial transactions, permissions, or other business-critical information.

5. Require human approval for sensitive actions

A dynamically generated interface can make it easier for an agent to request user confirmation.

However, the approval must be verified on the backend. Hiding or disabling a frontend button is not an authorization mechanism.

6. Handle interrupted streams and reconnection

Streaming applications need predictable recovery behavior.

Consider:

• Reconnecting after network failures
• Handling duplicate events
• Recovering from incomplete UI updates
• Preserving task and surface identifiers
• Handling cancellation
• Expiring stale sessions

AG-UI and A2UI define communication structures, but the application still needs an operational strategy for stream recovery and state consistency.

7. Audit agent-generated interactions

For enterprise applications, it should be possible to understand:

• Which agent generated a UI.
• Which components were requested.
• Which actions the user performed.
• Which backend operations were executed.
• Whether an approval was obtained.
• What the final result was.

This is particularly important when AI interfaces are used to interact with internal business systems.

14. What Can an AI Chapter Lead Contribute?

I believe this is where these technologies become particularly valuable.

The opportunity is not simply introducing generative UI into an application. It is establishing a reusable frontend-agent architecture that product teams can build on.

1. Establish a Shared Agent-Frontend Integration Layer

Instead of every team implementing its own event handling and agent-state synchronization, we can define a shared integration layer based on AG-UI.

It can provide:

• Event subscription and processing
• Streaming message handling
• Tool execution indicators
• State synchronization
• Cancellation and error handling
• Human approval workflows

This reduces duplicated infrastructure across applications.

2. Create a Governed Component Catalog

A2UI introduces an interesting opportunity to connect agent capabilities with an organization’s existing design system.

For example, we can define an approved component catalog:

ComponentPurposeSummaryCardDisplay structured summariesDataTableRender tabular informationLineChartDisplay time-series dataApprovalPanelPresent approval requestsDateRangePickerCollect date filtersStatusIndicatorDisplay execution status

The agent can request these components, while the frontend maps them to existing React components.

This means teams can introduce dynamic interfaces without giving the model unrestricted access to the frontend codebase.

For organizations with established design systems, this could be particularly valuable.

3. Define Reusable Frontend Patterns

We can establish standard patterns for:

• Agent progress indicators
• Interactive forms
• Approval interfaces
• Dynamic dashboards
• Multi-step workflows
• Tool execution views
• Subagent activity panels

These patterns can become shared components and examples that product teams reuse.

4. Standardize the Security Model

A shared approach should define which components agents can request, which actions require approval, and how the backend validates every operation.

This is especially important for applications interacting with sensitive business systems.

5. Build a Reference Implementation

A small reference application could demonstrate the complete interaction:

1. A user asks an agent to analyze a dataset.
2. The agent starts execution and sends AG-UI events.
3. The frontend displays progress and activity.
4. The agent generates an A2UI dashboard.
5. The React renderer maps the description to approved components.
6. The user modifies a filter or submits an action.
7. The frontend sends the interaction back to the agent.
8. The agent updates the interface with new results.

This would help developers understand how the protocols work together in a real application.

6. Measure the Developer and User Experience

We should not evaluate generative UI only by how quickly an agent can produce a visual result.

Useful metrics include:

• Time to interactive UI
• Agent-to-frontend event latency
• UI schema validation failure rate
• Unsupported component requests
• Rendering errors
• User interaction completion rate
• Human approval rate
• Frontend integration effort

These metrics can help determine whether generative UI actually improves the application experience.

15. What Is the Current State of A2UI and AG-UI?

Both projects are evolving, so protocol versioning matters.

As of November 1, 2026:

AG-UI

AG-UI publishes a versioned v1.0 specification and provides TypeScript client libraries, event schemas, and integrations with different agent frameworks.

Its ecosystem includes integrations with projects such as LangGraph, CrewAI, Google ADK, and other agent runtimes.

A2UI

A2UI currently identifies v0.9.1 as its production protocol release.

The v1.0 specification is a candidate release, not the production version.

The project also provides an official React renderer, shared web core, Lit renderer, Angular renderer, and Flutter-based support.

There is an important migration consideration: the v1.0 candidate introduces changes to the protocol model and should not be assumed to be wire-compatible with v0.9.1.

For production applications, I would start with the stable v0.9.1 protocol and its matching renderer, and evaluate v1.0 separately before planning a migration.

References: A2UI Repository and Status · A2UI Roadmap · AG-UI Specification

Conclusion

AG-UI and A2UI address two important challenges in building modern agent-powered applications.

AG-UI standardizes the interaction between agents and frontend applications through a structured, event-driven communication model.

A2UI allows agents to describe dynamic interfaces using declarative JSON messages, while applications remain in control of the actual components and rendering process.

Together, they make it possible to move beyond traditional chat interfaces and build applications where AI agents can communicate their progress, update application state, and generate context-aware user interfaces.

For engineering teams, the opportunity is to establish reusable interaction patterns, govern agent-accessible components, maintain security boundaries, and integrate generative UI into existing design systems.

My recommendation: Start with a small React application that uses AG-UI for streaming and state synchronization, then introduce A2UI for one clearly defined dynamic interface. Evaluate rendering reliability, security, and user experience before expanding the approach.

The goal should not be to make every screen dynamic.

It should be to make the interface adapt when a conversation or workflow genuinely requires it.

References and Further Reading

AG-UI

1. AG-UI Official Documentation — Introduction, architecture, events, and integration.
2. AG-UI GitHub Repository — Protocol implementation and SDKs.
3. AG-UI Architecture — Event-driven architecture and client-server communication.
4. AG-UI Event Reference — Lifecycle, messages, tools, and state events.
5. AG-UI TypeScript SDK — Client implementation and event handling.

A2UI

1. Google A2UI Official Documentation — Project overview and developer resources.
2. A2UI Protocol v0.9.1 — JSON protocol, surfaces, components, and data models.
3. A2UI Quickstart — Full-stack example and generated UI workflow.
4. A2UI React Renderer — React rendering implementation.
5. A2UI Client Setup — Framework integration and component catalogs.
6. A2UI Renderer Development — Building custom renderers and integrating design systems.
7. A2UI v1.0 Candidate Specification — Upcoming protocol capabilities and changes.

Related Technologies

1. Vercel AI SDK — useChat — Conversational streaming and frontend state.
2. Vercel — MCP Apps Integration — Interactive tool UIs in AI SDK applications.
3. CopilotKit — Shared State — Bidirectional state synchronization between React and agents.

Most services eventually need a concurrency limit.

You might start with something simple:

const maxConcurrentRequests = 100

A semaphore ensures that no more than 100 requests execute simultaneously.

It works. Until the conditions change.

Your service gets more CPU resources after scaling. A downstream database becomes slower. A new deployment changes request processing time. Traffic patterns shift throughout the day.

The limit that worked yesterday might be too restrictive today. Or worse, it might be too high and allow a dependency to become overloaded.

Static concurrency limits are easy to implement, but they require continuous tuning.

What if a service could adjust its concurrency limit automatically based on how the dependency behaves?

That’s the idea behind adaptive concurrency limiting.

Instead of relying entirely on a predefined number, the system observes request latency and failure signals, estimates when queueing is happening, and adjusts the number of requests it allows in flight.

In this article, we’ll explore:

• Why fixed concurrency limits eventually become difficult to maintain
• The relationship between concurrency, throughput, and latency
• How TCP congestion control inspired adaptive concurrency algorithms
• How to implement a latency-aware concurrency limiter in Go
• How to integrate it into an HTTP client
• How adaptive limits interact with backpressure, bulkheads, and retry budgets
• What to monitor and test before using adaptive concurrency in production

1. The Problem with Fixed Concurrency Limits

Imagine a Go service that calls an external market data API.

You initially configure a concurrency limit of 50.

limiter := NewFixedLimiter(50)

Under normal traffic, the dependency responds in 40ms.

Everything looks healthy.

Then traffic increases.

The dependency starts processing requests more slowly. Its response time grows to 150ms, then 300ms.

Your service still allows 50 concurrent requests.

The problem is that concurrency and latency are connected.

When requests take longer to complete, existing requests occupy concurrency slots for longer. New requests accumulate, queues grow, and latency increases further.

A fixed limit cannot automatically distinguish between:

• A dependency that has additional capacity
• A dependency approaching saturation
• A dependency already experiencing queueing
• A dependency recovering from an overload

You can increase the limit manually, but that risks overloading the dependency.

You can decrease it, but that might unnecessarily reduce throughput.

The challenge is finding a concurrency level that allows high throughput without creating excessive queueing.

2. Why Concurrency Matters More Than RPS Alone

Rate limiting and concurrency limiting solve different problems.

A rate limiter controls how many requests can start during a period.

A concurrency limiter controls how many requests can be in progress simultaneously.

Consider two situations:

ScenarioRequest rateAverage latencyApprox. concurrencyHealthy dependency500 req/s40ms20Slow dependency500 req/s200ms100

Both scenarios have the same incoming request rate.

But the second requires five times as many concurrent requests to sustain that throughput.

This relationship is described by Little’s Law:

[ L = \lambda W ]

Where:

• (L) is the average number of items in the system
• (\lambda) is the average arrival rate
• (W) is the average time spent in the system

For request processing:

[ Concurrency \approx Throughput \times Latency ]

This explains why a fixed RPS limit is not always sufficient to protect a dependency.

A service can receive an acceptable number of requests per second while still accumulating too many in-flight operations because response times have increased.

Adaptive concurrency focuses on this relationship.

3. The Idea Behind Adaptive Concurrency

The central idea is straightforward:

1. Measure request latency.
2. Maintain an estimate of normal, unloaded latency.
3. Compare current latency against that baseline.
4. Increase concurrency when the dependency has spare capacity.
5. Reduce concurrency when latency indicates queueing or failures.

This resembles congestion control in TCP.

TCP does not blindly send unlimited data. It maintains a congestion window and adjusts how much data can be in flight based on network conditions.

Adaptive concurrency applies a similar idea to service requests.

The concurrency limit becomes a dynamic window.

<figure> <img src=”https://dummyimage.com/1200x500/101827/00eaff.png&text=Adaptive+Concurrency%3A+Measure+%E2%86%92+Adjust+%E2%86%92+Protect" alt=”Adaptive concurrency control loop” /> </figure>

There are several algorithms worth understanding.

AIMD

Additive Increase, Multiplicative Decrease.

• Increase the limit gradually when requests succeed.
• Reduce it significantly when failures or overload signals appear.

For example:

Current limit: 50

Successful samples:
50 → 51 → 52 → 53

Overload detected:
53 → 37

AIMD is relatively simple and works well when the system has reliable failure signals.

However, it may react only after a dependency starts rejecting requests or timing out.

Vegas

Vegas uses latency as an early signal of congestion.

It estimates queueing using the difference between observed latency and a no-load baseline.

A simplified queue estimate is:

[ Q = L \times \left(1 — \frac{RTT{baseline}}{RTT{current}}\right) ]

Where:

• (Q) is estimated queue size
• (L) is the current concurrency limit
• (RTT_{baseline}) is the observed no-load latency
• (RTT_{current}) is the current measured latency

When latency stays close to the baseline, the algorithm can increase concurrency.

When latency grows significantly, it reduces the limit.

Gradient2

Gradient2 compares short-term and long-term latency averages to detect changes in the latency trend.

It is useful when latency baselines drift over time or workloads are bursty.

Instead of relying only on a minimum latency measurement, it uses smoothed averages to identify growing queueing pressure.

For a deeper implementation reference, Netflix’s concurrency-limits project is worth exploring:

https://github.com/Netflix/concurrency-limits

For this article, we’ll build a smaller Vegas-inspired controller. It is an educational implementation, not a complete reproduction of Netflix’s Vegas or Gradient2 algorithms.

4. Building an Adaptive Limiter in Go

Let’s implement a limiter that:

• Tracks in-flight requests
• Enforces a dynamic concurrency limit
• Measures request duration
• Collects latency samples in windows
• Estimates queueing using a baseline latency
• Increases or decreases the limit based on observed behavior

We’ll use a fail-fast admission strategy.

When the current concurrency limit is reached, the limiter rejects the new request instead of allowing an unbounded queue to form.

Step 1: Define the limiter

package adaptive

import (
    "errors"
    "sync"
    "time"
)

var ErrLimitExceeded = errors.New(
    "adaptive concurrency limit exceeded",
)

type Limiter struct {
    mu sync.Mutex
    limit    int
    minLimit int
    maxLimit int
    inflight int
    peak     int
    baseline time.Duration
    sampleCount int
    sampleTotal time.Duration
    dropped     bool
    windowSize int
    alpha float64
    beta  float64
}

func New(
    initialLimit int,
    minLimit int,
    maxLimit int,
) *Limiter {
    if minLimit < 1 ||
        initialLimit < minLimit ||
        maxLimit < initialLimit {
        panic("invalid concurrency limits")
    }

    return &Limiter{
        limit:      initialLimit,
        minLimit:   minLimit,
        maxLimit:   maxLimit,
        windowSize: 20,
        alpha:      2,
        beta:       4,
    }
}

The limiter maintains two important pieces of information:

Current limit: The maximum number of requests allowed simultaneously.

In-flight count: The number of requests currently executing.

We also maintain latency statistics and a baseline.

The windowSize determines how many completed requests we collect before updating the concurrency limit.

Using a window helps avoid changing the limit after every individual request.

Step 2: Acquire a permit

The limiter needs to decide whether a new request can execute.

func (l *Limiter) Acquire() (func(bool), error) {
    l.mu.Lock()

    if l.inflight >= l.limit {
        l.mu.Unlock()
        return nil, ErrLimitExceeded
    }

    l.inflight++

    if l.inflight > l.peak {
        l.peak = l.inflight
    }

    l.mu.Unlock()

    startedAt := time.Now()

    var once sync.Once

    done := func(dropped bool) {
        once.Do(func() {
            elapsed := time.Since(startedAt)
            l.mu.Lock()
            defer l.mu.Unlock()
            l.inflight--
            l.recordSample(elapsed, dropped)
        })
    }

    return done, nil
}

The returned function releases the permit and reports the outcome.

For example:

done, err := limiter.Acquire()
if err != nil {
    return err
}

defer done(false)

return callDependency()

If the operation experiences a dependency-level overload signal, we can report that separately:

done, err := limiter.Acquire()
if err != nil {
    return err
}

defer func() {
    done(isOverloadError(err))
}()

err = callDependency()

return err

The sync.Once protects against accidentally releasing the same permit twice.

The caller must invoke the returned function exactly once in normal control flow. The sync.Once makes repeated invocations harmless.

Step 3: Record latency samples

Now we need to collect completed request samples.

func (l *Limiter) recordSample(
    latency time.Duration,
    dropped bool,
) {
    l.sampleCount++
    l.sampleTotal += latency

    if dropped {
        l.dropped = true
    }
    if l.sampleCount < l.windowSize {
        return
    }
    average := l.sampleTotal / time.Duration(l.sampleCount)
    l.updateLimit(average)
    l.sampleCount = 0
    l.sampleTotal = 0
    l.dropped = false
    l.peak = 0
}

Instead of adjusting concurrency after every request, we wait until a complete sample window is collected.

This reduces sensitivity to individual latency spikes.

A production implementation may use more sophisticated rolling windows, exponential moving averages, or percentile-based measurements.

Step 4: Implement the control algorithm

Now comes the interesting part.

We compare the current average latency against the baseline.

func (l *Limiter) updateLimit(
    currentLatency time.Duration,
) {
    if currentLatency <= 0 {
        return
    }

    if l.baseline == 0 ||
        currentLatency < l.baseline {
        l.baseline = currentLatency
        return
    }
    if l.dropped {
        l.decrease()
        return
    }
    queueEstimate := float64(l.limit) *
        (1 - float64(l.baseline)/float64(currentLatency))
    // Increase only when the previous window
    // demonstrated meaningful concurrency usage.
    if l.peak*2 < l.limit {
        return
    }
    switch {
    case queueEstimate < l.alpha:
        l.increase()
    case queueEstimate > l.beta:
        l.decrease()
    }
}

The queue estimate is an approximation of the queueing pressure observed at the dependency.

When current latency is close to the baseline, the estimated queue remains small.

When current latency increases, the estimated queue grows.

We use two thresholds:

• alpha: Below this value, the dependency appears to have spare capacity.
• beta: Above this value, queueing pressure is considered too high.

The gap between the thresholds creates a stability region where the limit does not change.

This helps avoid unnecessary oscillation.

Step 5: Increase and decrease concurrency

The final methods enforce the configured boundaries.

func (l *Limiter) increase() {
    if l.limit < l.maxLimit {
        l.limit++
    }
}

func (l *Limiter) decrease() {
    next := int(float64(l.limit) * 0.7)
    if next < l.minLimit {
        next = l.minLimit
    }
    l.limit = next
}

The limiter uses:

• Additive increase: Grow gradually.
• Multiplicative decrease: Reduce more aggressively when congestion appears.

This is intentionally conservative.

Increasing the limit by one allows the dependency to demonstrate additional capacity without creating a large traffic spike.

Reducing the limit by 30% provides a faster response to overload.

These values are configuration choices, not universal production defaults.

Step 6: Expose the current limit

We should expose the current state for observability.

func (l *Limiter) Stats() (
    limit int,
    inflight int,
    baseline time.Duration,
) {
    l.mu.Lock()
    defer l.mu.Unlock()

    return l.limit, l.inflight, l.baseline
}

Now we can inspect the limiter without exposing its internal synchronization.

For example:

limit, inflight, baseline := limiter.Stats()

log.Printf(
    "adaptive limiter: limit=%d inflight=%d baseline=%s",
    limit,
    inflight,
    baseline,
)

5. Integrating the Limiter with an HTTP Client

Let’s apply the limiter to an external dependency.

type MarketDataClient struct {
    httpClient *http.Client
    limiter    *adaptive.Limiter
}

func (c *MarketDataClient) Fetch(
    ctx context.Context,
    symbol string,
) error {
    done, err := c.limiter.Acquire()
    if err != nil {
        return err
    }

    var dropped bool

    defer func() {
        done(dropped)
    }()
    req, err := http.NewRequestWithContext(
        ctx,
        http.MethodGet,
        "https://api.example.com/market/"+symbol,
        nil,
    )

    if err != nil {
        return err
    }
    resp, err := c.httpClient.Do(req)

    if err != nil {
        dropped = isDependencyOverload(err)
        return err
    }

    defer resp.Body.Close()

    if resp.StatusCode == http.StatusTooManyRequests ||
        resp.StatusCode >= http.StatusServiceUnavailable {
        dropped = true
        return fmt.Errorf(
            "market data dependency returned %d",
            resp.StatusCode,
        )
    }

    if resp.StatusCode < 200 || resp.StatusCode >= 300 {
        return fmt.Errorf(
            "unexpected status: %d",
            resp.StatusCode,
        )
    }

    return nil
}

The example classifies overload responses separately from other errors.

This distinction matters.

A validation error, for example, should not necessarily cause the adaptive limiter to reduce its concurrency limit.

Likewise, a network timeout might be an overload signal in one system but a network infrastructure issue in another.

The failure classifier should reflect the dependency’s actual behavior.

The HTTP client should also have appropriate timeouts and connection settings. An adaptive concurrency limiter is not a replacement for transport-level protection.

6. What Happens During an Overload?

Let’s walk through a simplified scenario.

Assume the limiter starts with a concurrency limit of 50.

The dependency initially responds in 40ms.

As traffic increases, the limiter observes the following:

StageAverage latencyConcurrency limitInitial40ms50Increased traffic45ms51Higher utilization55ms52Queueing begins120ms36Dependency recovers45msGradually increases

The important behavior is not the exact numbers.

The limiter detects that latency is increasing relative to the baseline and reduces the amount of work it allows in flight.

As the dependency recovers, the limiter can gradually explore additional capacity again.

This is a feedback control loop.

However, it is important to understand that the limiter does not instantly discover the perfect concurrency limit. It continuously estimates a useful operating point based on observed signals.

7. Adaptive Concurrency vs. Other Resilience Patterns

Adaptive concurrency is not a replacement for the patterns we’ve discussed in previous articles.

It complements them.

MechanismResponsibilityBackpressureControls incoming workLoad sheddingRejects work when capacity is exceededBulkheadIsolates resources and dependenciesRetry budgetLimits additional retry trafficAdaptive concurrencyAdjusts the concurrency ceiling based on observed conditions

Consider a service that communicates with three dependencies:

• Market Data API
• Portfolio Database
• Reporting Service

Each dependency has different latency characteristics and capacity.

Using one global concurrency limit would couple their behavior.

A better approach is to create an independent adaptive limiter for each dependency.

type DependencyLimiters struct {
    MarketData *adaptive.Limiter
    Portfolio  *adaptive.Limiter
    Reporting  *adaptive.Limiter
}

This gives each dependency its own feedback loop.

If the reporting service becomes slow, its concurrency limit can decrease without automatically reducing the capacity available to market data requests.

This is where adaptive concurrency and bulkheads work particularly well together.

8. Important Production Considerations

The example implementation demonstrates the core idea, but a production adaptive limiter needs additional safeguards.

Baseline latency can become stale

A latency baseline is only useful when it represents a reasonably unloaded system.

If the baseline is established while the dependency is already overloaded, the algorithm may incorrectly treat unhealthy latency as normal.

Production implementations should carefully manage baseline updates and resets.

Traffic must be representative

A dependency may expose different latency characteristics for different operations.

For example:

• Reading a cached market price
• Fetching historical prices
• Calculating a portfolio
• Executing a database aggregation

Combining all these operations into one latency signal can produce misleading results.

Consider separate limiters or traffic partitions when workloads have significantly different behavior.

The algorithm needs enough samples

Adaptive control depends on observations.

A low-traffic service might not generate enough samples to update its limit reliably.

A new instance also needs a warm-up strategy.

Slow start, conservative initial limits, and minimum sample requirements can help.

Avoid uncontrolled oscillation

A limiter that constantly increases and decreases its capacity can introduce instability.

Useful techniques include:

• Sampling windows
• Exponential smoothing
• Separate increase and decrease thresholds
• Bounded increase and decrease rates
• Minimum and maximum concurrency limits
• Jitter when multiple independent clients share a dependency

Local limits multiply across replicas

This is especially important in distributed deployments.

Imagine ten replicas, each with an adaptive concurrency limit of 100.

The dependency may receive up to 1,000 concurrent requests.

Each instance is making decisions based on its own observations. The aggregate behavior is not automatically coordinated.

Local adaptive concurrency is often sufficient for protecting individual instances and reacting to local conditions.

But strict global dependency quotas may require a shared dispatcher, centralized coordination, or provider-side quota enforcement.

Do not confuse concurrency with connection limits

An HTTP transport’s connection pool and an application’s concurrency limiter protect different resources.

A request may hold an application permit while waiting for a connection. Conversely, HTTP/2 can multiplex multiple requests over one connection.

Configure both based on the actual workload.

9. Testing Adaptive Concurrency

Adaptive algorithms should be tested as control systems, not only as ordinary utility functions.

At minimum, test these behaviors:

• The initial concurrency limit is respected.
• Requests are rejected when the current limit is reached.
• Permits are released exactly once.
• Successful low-latency windows allow gradual growth.
• Rising latency causes the limit to decrease.
• Overload signals trigger a decrease.
• The limit never exceeds its configured maximum.
• The limit never falls below its configured minimum.
• Concurrent acquisitions are race-safe.
• Insufficient sample windows do not trigger premature changes.

Run the race detector:

go test -race ./...

Also consider deterministic algorithm tests.

Instead of using real HTTP requests and sleeping for arbitrary durations, separate the control algorithm from the admission mechanism.

Feed known latency samples into the controller and verify the resulting limit.

For example:

Baseline: 40ms

Window 1: 42ms → increase
Window 2: 45ms → increase
Window 3: 120ms → decrease
Window 4: 130ms → decrease
Window 5: 45ms → gradual recovery

This makes algorithm behavior easier to reproduce and debug.

Finally, run load tests with changing dependency latency.

A useful test should simulate:

1. A healthy dependency.
2. Gradually increasing latency.
3. Sudden overload.
4. Partial recovery.
5. A complete dependency outage.

Measure throughput, p95/p99 latency, rejection rate, and concurrency limit changes.

The goal is not simply to maximize throughput.

The goal is to maintain useful throughput without allowing latency and queueing to grow uncontrollably.

10. Observability

An adaptive limiter introduces a new operational component.

You need visibility into how it makes decisions.

Useful metrics include:

MetricTypePurposeadaptive_concurrency_limitGaugeCurrent concurrency ceilingadaptive_inflight_requestsGaugeActive requestsadaptive_request_latency_secondsHistogramObserved dependency latencyadaptive_rejected_totalCounterRequests rejected at admissionadaptive_limit_increase_totalCounterLimit increase eventsadaptive_limit_decrease_totalCounterLimit decrease eventsadaptive_baseline_latency_secondsGaugeEstimated unloaded latency

These metrics help answer important questions:

• Is the limiter constantly operating at its maximum?
• Is the concurrency limit frequently decreasing?
• Is the dependency latency increasing?
• Are requests being rejected before reaching the dependency?
• Is the limiter recovering too slowly?

A particularly useful dashboard plots three signals together:

1. Current concurrency limit
2. In-flight requests
3. Dependency latency

Looking at them independently can hide the relationship between capacity and latency.

Avoid high-cardinality labels such as raw URLs, user IDs, and request IDs.

11. When Should You Use Adaptive Concurrency?

Adaptive concurrency is not necessary for every service.

A fixed limit is often the better choice when:

• Dependency capacity is stable and well understood.
• Traffic patterns are predictable.
• The service has low request volume.
• The cost of an adaptive control loop is not justified.
• A simple semaphore already provides sufficient protection.

Adaptive concurrency becomes more attractive when:

• Dependency capacity changes frequently.
• Traffic is bursty or unpredictable.
• Services scale horizontally.
• Latency is a reliable indicator of queueing.
• Manual concurrency tuning is difficult.
• You need to protect dependencies before they start rejecting requests.

Start with a fixed limit if it is sufficient. Introduce adaptive behavior when operational evidence shows that static configuration is becoming a limitation.

Conclusion

A fixed concurrency limit assumes that system capacity is relatively predictable.

In distributed systems, that assumption does not always hold.

Dependencies slow down, traffic changes, instances scale, and resource availability evolves.

Adaptive concurrency provides a feedback loop that adjusts the amount of work allowed in flight based on observed latency and failure signals.

In Go, the core mechanism is straightforward: track active requests, collect latency samples, estimate congestion, and adjust the concurrency ceiling.

The difficult part is not implementing a semaphore.

It is building a stable controller that reacts to real overload without constantly oscillating or making incorrect decisions from noisy measurements.

Don’t just limit how much work your service can handle. Let it learn when it should slow down.

Understanding how AI agents discover, communicate, delegate tasks, and collaborate across different frameworks and platforms.

1. What Is A2A, and What Problem Does It Solve?

As AI applications become more capable, we are moving beyond simple interactions between a user and a single AI model.

Today, an AI agent can search documentation, query databases, interact with APIs, analyze code, and execute multi-step workflows. However, as these capabilities grow, a new architectural question emerges:

How can independently developed AI agents communicate and work together?

Consider an application that needs to analyze a GitLab merge request. The task may involve several different responsibilities:

• A security agent reviews potential vulnerabilities.
• A testing agent analyzes test coverage and identifies missing cases.
• An architecture agent evaluates design decisions and code structure.
• A review agent combines the findings and prepares a final report.

Each agent might use a different framework, model provider, toolset, or programming language.

Without a standardized communication mechanism, integrating these agents usually requires custom APIs, shared data contracts, and application-specific orchestration logic.

This is where Agent2Agent (A2A) Protocol becomes relevant.

A2A is an open protocol designed to enable communication and collaboration between independent AI agent systems. It provides a common interaction model regardless of the framework, programming language, or vendor used to build an agent.

The protocol was originally developed by Google and contributed to the Linux Foundation. It is designed around interoperability without requiring agents to expose their internal reasoning, memory, tools, or implementation details.

The important distinction is that A2A does not define how an agent thinks or performs its internal tasks. It defines how one agent can communicate with another.

In other words, an agent can be implemented using LangGraph, Google ADK, CrewAI, Semantic Kernel, or a completely custom solution and still communicate with another compatible agent through A2A.

Reference: A2A Protocol — Official Documentation

2. Why Do We Need A2A?

Most agent frameworks provide mechanisms for calling tools, delegating work, or coordinating multiple agents.

However, these mechanisms are generally designed around the framework’s own execution model.

For example, an agent built with one framework may have its own task execution logic, state management, and internal tool definitions. Connecting it directly to an agent developed using another framework can require custom integration work.

A2A addresses the interoperability problem.

Instead of requiring agents to understand each other’s internal implementations, they communicate through a standardized protocol.

This gives us several important capabilities:

• Discover agents and understand their capabilities.
• Delegate tasks to remote agents.
• Exchange messages and structured data.
• Track the lifecycle of long-running tasks.
• Receive progress updates through streaming.
• Exchange files and generated artifacts.
• Support authentication and authorization across agent boundaries.

A2A is particularly useful when agents belong to different teams, applications, organizations, or technology stacks.

The goal is not to turn every agent into the same implementation. The goal is to allow different implementations to collaborate.

3. A2A vs. MCP vs. ACP

These three protocols are often discussed together, but they solve different architectural problems.

TechnologyMain responsibilityMCPAgent ↔ Tools and external contextACPCoding Agent ↔ Client ApplicationA2AAgent ↔ Agent

Let’s consider a practical example.

Imagine a research agent that needs to investigate a topic, retrieve information from internal documentation, and delegate a part of its work to a specialized analysis agent.

In this scenario:

1. MCP provides access to documentation, APIs, databases, and other tools.
2. A2A allows the research agent to communicate with the analysis agent.
3. ACP could be used separately when integrating a coding agent with an editor or another client application.

A simplified architecture:

                    User
                     |
                     v
              Main AI Application
                     |
                     v
               Coordinator Agent
                 /           \
                /             \
             A2A               A2A
              /                 \
             v                   v
      Research Agent       Analysis Agent
             |                   |
            MCP                 MCP
             |                   |
       Internal APIs        Database / APIs
             |                   |
             v                   v
          Results             Results
                \             /
                 \           /
                  Coordinator
                       |
                       v
                 Final Response

Each agent can use its own tools and internal architecture. A2A provides the communication layer between them.

One important point: A2A is not a replacement for an agent framework or an internal sub-agent mechanism. It focuses on communication between independent agent applications. Internal orchestration can still be handled by the framework that runs the agent.

References: A2A and MCP · MCP Documentation · ACP Documentation

4. How Does A2A Work?

To understand A2A, we need to understand a few core concepts.

4.1. Agent Card: Discovering Other Agents

One of the most important concepts in A2A is the Agent Card.

An Agent Card is a JSON document that describes an agent and its capabilities.

It acts as a machine-readable description of what an agent can do and how another agent can communicate with it.

An Agent Card can contain:

• Agent name and description
• Service endpoint
• Supported protocol versions and transports
• Available capabilities
• Supported input and output formats
• Skills and example tasks
• Authentication requirements

By default, an A2A server exposes its Agent Card at:

https://agent.example.com/.well-known/agent-card.json

For example, a research agent might expose the following description:

{
  "name": "Research Agent",
  "description": "An agent specialized in research and source analysis.",
  "version": "1.0.0",
  "supportedInterfaces": [
    {
      "url": "https://research.example.com/a2a",
      "protocolBinding": "JSONRPC",
      "protocolVersion": "1.0"
    }
  ],
  "capabilities": {
    "streaming": true,
    "pushNotifications": true,
    "extensions": []
  },
  "defaultInputModes": [
    "text/plain"
  ],
  "defaultOutputModes": [
    "text/plain",
    "application/json"
  ],
  "skills": [
    {
      "id": "market-research",
      "name": "Market Research",
      "description": "Researches topics and produces structured findings.",
      "tags": [
        "research",
        "analysis"
      ],
      "examples": [
        "Research the latest developments in AI agents."
      ],
      "inputModes": [
        "text/plain"
      ],
      "outputModes": [
        "text/plain",
        "application/json"
      ]
    }
  ]
}

This is a simplified public Agent Card example. Authentication and additional security requirements can also be described through the protocol’s security fields.

A client can retrieve this document and understand whether the remote agent is suitable for a particular task.

For example, a coordinator agent looking for an agent capable of analyzing financial documents can inspect registered Agent Cards and evaluate their declared skills.

Agent discovery can happen through:

• Well-known URLs
• Private or public agent registries
• Directly configured Agent Card URLs

An Agent Card describes an agent’s advertised capabilities; it does not prove that the agent can successfully perform every advertised task. Capability selection should be combined with authorization, availability checks, and runtime evaluation.

Reference: A2A — Agent Discovery

4.2. Messages and Tasks

A2A distinguishes between a message and a task.

Message

A message represents communication between a client and an agent. It can contain instructions, clarification requests, or additional information.

Task

A task represents a stateful unit of work with its own lifecycle.

For example, when an agent receives a request to analyze a large document, it may need several minutes to complete the work. Instead of treating the interaction as a simple request-response operation, A2A allows the agent to create a task and report its progress.

A task has a lifecycle that can include:

StateMeaningSubmittedTask has been acceptedWorkingAgent is processing the taskInput RequiredAdditional information is neededAuth RequiredAdditional authentication is requiredCompletedTask finished successfullyFailedTask execution failedCanceledTask was canceledRejectedAgent declined the task

Some states represent interruptions that can be resolved through further interaction, while others represent terminal states.

This is important for long-running workflows because the client does not need to keep a single request open until every operation is complete.

A2A also provides two useful identifiers:

• taskId: Identifies an individual task.
• contextId: Groups related messages and tasks within a broader interaction context.

The task identifier is generated by the agent server. The context identifier helps maintain continuity across multiple related interactions.

4.3. Artifacts: Sharing Task Results

A2A also introduces the concept of Artifacts.

An artifact represents the output produced by an agent while completing a task.

For example:

• A research agent can produce a research report.
• A coding agent can return a generated file.
• An analysis agent can return structured JSON.
• A document processing agent can return an extracted document.

An artifact can contain multiple parts, including text, binary content, URLs, or structured data.

This distinction is important:

Messages are primarily used for communication and interaction, while artifacts represent the actual outputs of a task.

It allows agents to exchange more than just text-based responses.

4.4. Streaming and Asynchronous Communication

Not every agent task finishes immediately.

A research process, document analysis, or code review can take a considerable amount of time.

A2A supports multiple ways of receiving task updates:

Streaming

The client can subscribe to task updates and receive events while the agent is working.

Depending on the protocol binding, streaming is supported through Server-Sent Events (SSE) or gRPC streaming.

Polling

The client can periodically request the current task state.

Push Notifications

For long-running tasks where maintaining a persistent connection is not practical, the agent can send updates to a client-provided webhook.

These mechanisms allow applications to support different execution patterns without inventing custom communication protocols for each agent.

A2A also defines operations for sending messages, retrieving tasks, canceling tasks, subscribing to updates, and managing push notification configurations.

References: A2A Specification · Task Lifecycle · Streaming and Asynchronous Operations

5. How to Integrate A2A?

A2A provides official SDKs for several programming languages, including:

• Python
• JavaScript / TypeScript
• Java
• Go
• C# / .NET
• Rust

For TypeScript applications, the official package is @a2a-js/sdk.

The current SDK implements the A2A v1.0 specification and supports client and server development.

Step 1: Install the package

pnpm add @a2a-js/sdk

The official JavaScript SDK requires Node.js 20 or later.

For this example, we will implement a client that communicates with an existing remote research agent.

Step 2: Connect to an A2A agent

The SDK provides ClientFactory to create a client.

import { ClientFactory } from '@a2a-js/sdk/client';

const factory = new ClientFactory();

const client = await factory.createFromUrl(
  'https://research-agent.example.com'
);

The SDK retrieves the Agent Card and selects a compatible transport based on the agent’s advertised interfaces and the client’s supported transports.

This means we do not need to hardcode every protocol detail into our application.

Step 3: Send a message to the agent

Once the connection is established, we can send a task request.

import {
  Role,
  type Message,
  type SendMessageRequest,
} from '@a2a-js/sdk';

const message: Message = {
  messageId: crypto.randomUUID(),
  role: Role.ROLE_USER,
  parts: [
    {
      content: {
        $case: 'text',
        value: 'Research the latest developments in multi-agent systems.',
      },
      mediaType: 'text/plain',
      filename: '',
      metadata: undefined,
    },
  ],
  taskId: '',
  contextId: '',
  extensions: [],
  metadata: {},
  referenceTaskIds: [],
};

const request: SendMessageRequest = {
  tenant: '',
  message,
  configuration: undefined,
  metadata: {},
};

const result = await client.sendMessage(request);

console.log(result);

The example uses the SDK’s generated TypeScript message types. Depending on the agent’s behavior, the response can contain a message or a task.

For production integrations, authentication and request timeouts should be configured as well.

For example:

const result = await client.sendMessage(request, {
  serviceParameters: {
    Authorization: `Bearer ${process.env.A2A_TOKEN}`,
  },
  signal: AbortSignal.timeout(60_000),
});

The client can now communicate with a remote agent without needing to know its internal framework or model implementation.

Step 4: Stream task updates

For long-running tasks, we can use sendMessageStream.

const stream = client.sendMessageStream(request);

for await (const event of stream) {
  // Handle task events:
  // - task
  // - statusUpdate
  // - artifactUpdate
  console.log(event);
}

The actual event union should be handled using the SDK’s generated types.

This allows us to display task progress, process generated artifacts, and update the application as the remote agent works.

The SDK also supports task retrieval, cancellation, and push notification configuration.

Step 5: Build your own A2A agent

A2A is not limited to consuming existing agents. We can also expose our own agent through the protocol.

The official JavaScript SDK separates server-side responsibilities into three main components:

AgentExecutor

Contains the agent’s business logic. It receives the request context, executes the task, and publishes messages, status updates, and artifacts.

DefaultRequestHandler

Handles protocol-level responsibilities such as message routing, task storage, cancellation, and push notifications.

Transport Adapters

Expose the agent through supported communication protocols.

The SDK provides adapters for JSON-RPC and HTTP+JSON/REST through Express, as well as gRPC for Node.js.

The architecture looks like this:

Incoming A2A Request
        |
        v
 Transport Adapter
        |
        v
DefaultRequestHandler
        |
        +----> Task Store
        |
        +----> Execution Event Bus
        |
        v
   AgentExecutor
        |
        v
   Agent Logic
        |
        +----> LLM
        +----> MCP Tools
        +----> Internal APIs
        +----> Business Logic
        |
        v
 Messages / Status / Artifacts
        |
        v
   A2A Response

The SDK also supports custom task stores and event buses, which allows us to integrate persistent databases, caches, or message brokers instead of relying exclusively on in-memory implementations.

For production, this separation is particularly useful because protocol handling, task execution, and persistence can evolve independently.

Reference: Official A2A JavaScript SDK

6. Integrating A2A with the Vercel AI SDK

There is another interesting integration worth looking at.

The Vercel AI SDK lists an A2A community provider called a2a-ai-provider.

This package allows an A2A-compatible agent to be consumed through familiar AI SDK functions such as generateText and streamText.

This is conceptually similar to the ACP provider we discussed previously.

Instead of implementing every A2A interaction manually, we can use the community provider as an adapter between the A2A protocol and the AI SDK.

Installation

pnpm add ai a2a-ai-provider

Provider configuration

import { a2a } from 'a2a-ai-provider';
import { generateText } from 'ai';

const result = await generateText({
  model: a2a(
    'https://research-agent.example.com/.well-known/agent-card.json'
  ),
  prompt: 'Research the latest developments in agentic AI.',
});

console.log(result.text);

Streaming responses

We can also use streamText:

import { a2a } from 'a2a-ai-provider';
import { streamText } from 'ai';

const result = streamText({
  model: a2a(
    'https://research-agent.example.com/.well-known/agent-card.json'
  ),
  prompt: 'Analyze the current multi-agent ecosystem.',
  providerOptions: {
    a2a: {
      contextId: crypto.randomUUID(),
    },
  },
});

await result.consumeStream();

console.log(await result.content);

Here, contextId is used to associate related interactions with the same context.

This integration makes it easier to use A2A agents in existing AI SDK applications.

However, there is an important distinction:

The official A2A SDK is the protocol implementation, whereas a2a-ai-provider is a community-maintained adapter.

The community provider’s documentation describes it as under active development, and its GitHub repository currently labels it alpha and advises against production workloads.

For that reason, I would use it for experimentation and evaluate the official SDK directly for production-critical integrations until the provider’s maturity and compatibility are established.

References: AI SDK — A2A Community Provider · a2a-ai-provider GitHub

7. Supported Communication Protocols

A2A separates its common data model and operations from the transport used to communicate.

The current specification defines three standard protocol bindings:

ProtocolDescriptionJSON-RPCJSON-based remote procedure callsHTTP+JSON / RESTResource-oriented HTTP communicationgRPCRPC communication using Protocol Buffers and gRPC

An agent can support more than one binding.

For example, a service could expose JSON-RPC and REST interfaces using the same underlying task execution logic.

This separation is valuable because clients and servers can select a compatible transport without changing the fundamental meaning of tasks, messages, and artifacts.

The official JavaScript SDK supports all three transports, with its gRPC integration requiring a Node.js environment and additional dependencies.

A2A also defines protocol versioning. Clients and agents use the A2A-Version header to identify the version they are communicating with.

The current v1.0 specification includes changes from the earlier v0.3 interface, so compatibility and migration should be considered when connecting existing implementations.

Reference: A2A Protocol Specification

8. Advantages of A2A

Interoperability across frameworks

One of A2A’s biggest advantages is the ability to connect agents built using different frameworks and programming languages.

We do not have to standardize every team’s internal agent implementation just to enable collaboration.

A Python-based research agent can communicate with a TypeScript-based coordinator, while another agent may be implemented using Go or .NET.

This reduces framework-level coupling.

Independent agent implementations

Agents do not need to expose their internal prompts, model configurations, memory, or tools.

They only need to expose their supported capabilities and implement the agreed communication contract.

This allows teams to evolve their internal implementations without forcing all consumers to change their integration code.

Better support for long-running workflows

A2A is designed around stateful tasks.

Instead of implementing custom job tracking, status polling, event streaming, and result retrieval for every remote agent, we can use standardized operations.

This is particularly useful for research, document processing, code analysis, and other workflows that may take several minutes or longer.

More flexible output handling

Agents can exchange text, files, structured data, and other supported content types through message parts and artifacts.

This makes the protocol useful beyond conventional text-based chat applications.

Centralized agent discovery and integration

Agent Cards allow applications to discover what an agent offers and how to communicate with it.

Combined with a curated internal agent registry, this can provide a common discovery layer for enterprise AI platforms.

9. Limitations and Trade-offs

A2A solves a real interoperability problem, but it also introduces architectural complexity.

It does not provide orchestration by itself

A2A defines how agents communicate, not how an entire multi-agent system should be planned and executed.

It does not decide which agent should receive a task, how many agents should participate, or how conflicting results should be resolved.

Those responsibilities remain with an orchestrator or an agent framework.

For example, LangGraph, Google ADK, or a custom coordinator can decide when to delegate a task through A2A.

Distributed execution introduces more failure scenarios

When agents communicate across network boundaries, new failure scenarios appear:

• A remote agent may become unavailable.
• A request may time out while the task continues running.
• A task may fail after partial execution.
• A client may disconnect before receiving the final artifact.
• A downstream agent may return an incomplete or unexpected result.

Therefore, retries, cancellation, idempotency, timeout policies, and persistent task storage need to be considered carefully.

More communication can increase latency and cost

Every agent handoff introduces additional work.

A complex multi-agent workflow may involve several model calls, network requests, and tool executions.

Using more agents does not automatically improve the result.

We should introduce agent delegation only when the additional specialization or parallelism justifies the extra latency, cost, and operational complexity.

Agent compatibility does not guarantee equivalent capabilities

Two agents may both implement A2A but provide very different skills and execution behavior.

A2A standardizes communication, not the quality or accuracy of an agent.

Clients still need to evaluate supported capabilities, input/output formats, and the reliability of the remote service.

Version compatibility requires attention

The transition from earlier protocol versions to v1.0 introduced changes to operation names, Agent Cards, and other protocol details.

The official TypeScript SDK provides an opt-in v0.3 compatibility layer, which can help during staged migration.

However, version compatibility should be tested explicitly rather than assumed.

Security does not come automatically

A2A supports authentication and authorization mechanisms, but the protocol does not automatically make remote agents trustworthy.

The application must decide which agents can communicate, what data can be shared, and which actions are permitted.

Reference: A2A v1.0 Specification · A2A Security Considerations

10. Security and Production Considerations

This is one of the most important parts of adopting A2A in an enterprise environment.

An agent may receive data from another agent, process documents, invoke tools, and return results that influence subsequent decisions.

We should not treat every remote agent as trusted simply because it implements the same protocol.

Some important considerations:

Authentication

A2A relies on established authentication mechanisms such as HTTP authentication, OAuth 2.0, OpenID Connect, API keys, and mutual TLS where appropriate.

The agent can advertise supported authentication schemes through its Agent Card.

Credentials are handled at the transport layer, not embedded directly into protocol messages.

Authorization

Authentication and authorization are different responsibilities.

Knowing who is calling an agent does not automatically mean that the caller should be allowed to invoke every skill.

Authorization should consider:

• Which application or user initiated the task?
• Which agent is allowed to communicate with the target agent?
• Which skills can be invoked?
• Which business operations are permitted?
• Which data can be accessed or shared?

For financial or other sensitive enterprise environments, authorization should be enforced at the skill and business-action levels.

Data isolation

Agents should receive only the information required to complete their tasks.

A research agent should not automatically receive the entire conversation history, database access, or credentials of another agent.

Task and context storage should also respect user and tenant boundaries.

Agent Card validation

Agent Cards describe service identity and capabilities.

For sensitive environments, clients should verify the origin of Agent Cards and validate their signatures when available.

A discovery result should not automatically be treated as a trusted execution target.

Observability and auditing

A multi-agent system needs visibility beyond individual model calls.

We should record:

• Caller and target agent identifiers
• Task and context identifiers
• Correlation and trace identifiers
• Task state transitions
• Request duration and failure reasons
• Artifact metadata and access
• Sensitive action approvals

OpenTelemetry and distributed tracing can help follow a request across multiple agents.

Push notification security

Push notification webhooks need their own security controls.

For example, the receiver should authenticate the notification source, validate the task identifier, and process duplicate notifications safely.

Webhook endpoints should also be protected against SSRF and unauthorized access.

These controls are application and infrastructure responsibilities that must be designed alongside the protocol integration.

Reference: A2A — Enterprise Implementation

11. What Can an AI Chapter Lead Contribute?

I think this is where A2A becomes especially interesting from an engineering leadership perspective.

The value is not simply adding another protocol to the technology stack. It is establishing a reusable architecture for teams that want to build specialized agents independently and allow them to collaborate.

There are several areas where we can contribute.

1. Establish a common agent communication standard

Different teams may build agents using different languages and frameworks.

Instead of forcing everyone to adopt the same framework, we can define a common communication contract based on A2A.

This allows teams to own their agent implementations while maintaining a consistent integration approach.

2. Build an internal agent registry

We can create a centralized catalog of available agents and their capabilities.

For example:

AgentResponsibilityAvailable skillsResearch AgentInformation gatheringResearch, source analysisDatabase AgentData analysisSQL analysis, reportingCode Review AgentCode qualityReview, recommendationsSecurity AgentSecurity analysisVulnerability assessmentDocumentation AgentDocumentationSearch, summarization

Each agent can publish an Agent Card, while the internal registry handles discovery, ownership, versioning, authorization, and availability.

This would help teams reuse existing capabilities instead of implementing the same functionality repeatedly.

3. Define a reusable orchestration architecture

A2A should be combined with a clear orchestration strategy.

We can establish a reference architecture that separates:

• Agent discovery
• Task delegation
• Agent communication
• Execution state management
• Tool and data access
• Result aggregation
• Human approval

The coordinator should remain responsible for task planning and result aggregation. Specialized agents should focus on their assigned responsibilities.

4. Establish security and governance standards

A2A introduces communication between independently managed systems.

We should define clear standards for agent authentication, authorization, data sharing, tenant isolation, and audit logging.

This is especially important when agents interact with internal systems or execute business operations.

5. Run a practical multi-agent pilot

Rather than introducing A2A across every team immediately, we can start with a well-defined workflow.

A good example would be an automated merge request review process.

Example workflow:

1. A developer submits a merge request.
2. The Coordinator Agent receives the review request.
3. The Security Agent analyzes potential vulnerabilities.
4. The Testing Agent evaluates test coverage and identifies missing cases.
5. The Architecture Agent reviews design and implementation decisions.
6. Each specialist returns findings through A2A.
7. The Coordinator combines the results into a structured review report.
8. A developer reviews the recommendations before any changes are merged.

Each agent can use its own MCP servers to access GitLab, documentation, test tools, or other required services.

This allows us to evaluate the actual value of agent specialization and interoperability.

6. Measure the results

A successful pilot should be evaluated using real measurements.

Some useful metrics:

• Task completion rate
• Agent-to-agent communication latency
• Overall workflow duration
• Failed or timed-out tasks
• Duplicate execution rate
• Token and infrastructure costs
• Human intervention rate
• Quality of the final aggregated result

The objective is to determine whether distributing work across specialized agents produces better results than a single-agent implementation.

7. Create reusable learning resources

A practical reference implementation, architecture documentation, and internal workshops can help developers understand when A2A is useful.

Teams should learn not only how to implement the protocol, but also when not to use it.

Not every internal function needs to become a remote agent. For simple operations within the same runtime, a regular function or framework-native tool call may be more appropriate.

12. When Should We Use A2A?

A2A is most useful when communication between independent agents is an actual architectural requirement.

ScenarioRecommendationAgents built with different frameworks need to collaborateA2ADifferent teams expose independently maintained agentsA2ARemote, long-running agent tasks need trackingA2AAgents need to exchange structured outputs and filesA2AAn agent needs access to APIs and databasesMCPA single agent needs internal tool executionNative tools or MCPMultiple agents operate within one tightly coupled workflowFramework-native orchestration may be sufficient

The important thing is to avoid introducing network boundaries and protocol complexity when a simpler implementation would work.

A2A should solve an interoperability requirement, not become an additional abstraction by default.

Conclusion

A2A addresses an important challenge in the evolution of agentic AI systems: enabling independently developed agents to communicate and collaborate.

As AI applications become more specialized, a single agent will not necessarily be responsible for every task. Different agents may own different capabilities, use different frameworks, and operate in different environments.

A2A provides a standardized way for these systems to discover each other, delegate tasks, exchange information, and track execution.

Combined with MCP, it creates a useful architectural separation: MCP connects agents to tools and external context, while A2A connects agents to other agents.

For an AI Chapter Lead, the opportunity is to establish common integration standards, build an internal agent ecosystem, define security boundaries, and help teams develop reusable capabilities.

My recommendation: Start with one practical multi-agent workflow, implement it using the official A2A SDK, and evaluate the operational benefits against a conventional single-agent or framework-native orchestration approach.

The real question is not how many agents we can connect. It is whether connecting specialized agents makes the system more reliable, maintainable, and effective.

References and Further Reading

1. A2A Protocol — Official Website — Overview, architecture, ecosystem, and documentation.
2. A2A Protocol — Specification v1.0 — Protocol operations, data models, task lifecycle, transports, and security.
3. A2A — Agent Discovery — Agent Cards, discovery mechanisms, and capability descriptions.
4. A2A — Life of a Task — Task lifecycle and interaction patterns.
5. A2A — Streaming and Asynchronous Operations — Streaming, task updates, and push notifications.
6. A2A — Enterprise Implementation — Authentication, authorization, observability, and enterprise security.
7. A2A — Official JavaScript SDK — TypeScript client/server implementation, transport adapters, and runnable examples.
8. A2A — Official GitHub Organization — SDKs, samples, and project development.
9. AI SDK — A2A Community Provider — Integrating A2A agents with Vercel AI SDK.
10. A2A AI Provider — Community provider implementation and current maturity status.
11. Model Context Protocol — Complementary agent-to-tool integration protocol.

Retries are one of the most common ways to make distributed systems more resilient.

A request fails because of a temporary network issue. The client waits briefly and tries again. The second attempt succeeds, and the failure never reaches the user.

That sounds reasonable.

But what happens when the dependency is already overloaded?

Requests start failing. Every caller retries. Those retries create additional traffic, which causes more failures, which trigger even more retries.

A temporary slowdown can turn into a retry storm.

Retries are not free. Every attempt consumes resources, occupies connections, increases queue pressure, and adds work to an already struggling dependency.

This is where retry budgets become useful.

A retry budget places a limit on how much additional traffic a service can generate through retries. Instead of allowing every failed request to retry independently, it establishes a shared budget across requests.

In this article, we’ll explore:

• Why per-request retry limits are not enough
• How retry amplification creates cascading failures
• The difference between retry limits and retry budgets
• How to implement a concurrency-safe retry budget in Go
• How to combine budgets with exponential backoff and jitter
• How retries interact with timeouts, bulkheads, and circuit breakers
• What to measure and test before using retries in production

1. The Problem: Retries Can Amplify Failures

Imagine a service that sends requests to an external market data provider.

Under normal conditions:

• The service receives 1,000 requests per second.
• Each request makes one provider call.
• The provider responds successfully.

Now the provider starts experiencing problems. Its error rate increases, and requests become slower.

The service retries failed requests up to three times.

If every original request fails, the provider may receive:

AttemptRequestsOriginal requests1,000First retries1,000Second retries1,000Third retries1,000Total4,000

The service has turned 1,000 incoming requests into as many as 4,000 downstream attempts.

This is retry amplification.

The actual situation can be worse. Retries may overlap with new incoming traffic, other services may retry independently, and clients may have their own retry policies.

The dependency is already struggling, but the system keeps sending it more work.

<figure> <img src=”https://dummyimage.com/1200x520/101827/00eaff.png&text=Retry+Amplification%3A+1000+Requests+%E2%86%92+4000+Attempts" alt=”Retry amplification diagram” /> </figure>

The important distinction is that retries can improve the success rate of individual requests while making the overall system less stable.

A retry policy should not only ask:

How many times should this request retry?

It should also ask:

How much additional traffic can this service afford to generate?

2. Why Max Retries Is Not Enough

A typical retry implementation has a maximum attempt count:

for attempt := 1; attempt <= 3; attempt++ {
    err := callDependency()
    if err == nil {
        return nil
    }
}

This limits retries for an individual request.

But it does not limit retries across the entire service.

Consider a service receiving 10,000 requests per second. Even with only one retry per request, it can generate another 10,000 downstream attempts per second during an outage.

A per-request retry limit provides a local constraint, not a global traffic constraint.

There are two separate controls:

ControlWhat it limitsMaximum attemptsRetries for one requestRetry budgetRetries across a service or dependency

You generally want both.

A maximum attempt count prevents one request from retrying indefinitely. A shared retry budget prevents thousands of requests from independently exhausting a dependency.

3. What Is a Retry Budget?

A retry budget is a shared allowance for additional attempts.

Instead of allowing every failed request to retry automatically, the service must acquire permission from the budget before performing another attempt.

One practical approach is a token bucket.

The model is simple:

1. Successful initial requests add tokens to the budget.
2. Each retry consumes one token.
3. The budget has a maximum capacity.
4. When no tokens are available, additional retries are rejected.

For example, suppose a service uses these settings:

• Maximum budget: 20 tokens
• Credit per successful initial request: 0.1 tokens
• Cost per retry: 1 token

Every ten successful initial requests generate one retry token.

If the service experiences a sudden failure, it can spend its accumulated tokens on retries. Once the budget is exhausted, it stops generating additional retry traffic until successful initial requests replenish it.

This creates an important feedback mechanism:

• Healthy service: successful requests replenish the budget.
• Unhealthy service: fewer successful requests mean less budget replenishment.
• Outage: retries consume the remaining budget and eventually stop.

The budget is not a guarantee that a dependency can handle a particular amount of traffic. It is a mechanism for limiting retry amplification.

4. Implementing a Retry Budget in Go

Let’s start with the shared budget.

We need a structure that is safe to access concurrently because multiple requests may attempt to consume or replenish tokens at the same time.

package retry

import "sync"

type RetryBudget struct {
    mu sync.Mutex
    tokens          float64
    capacity        float64
    creditPerSuccess float64
}

func NewRetryBudget(
    capacity float64,
    creditPerSuccess float64,
) *RetryBudget {
    return &RetryBudget{
        capacity:         capacity,
        creditPerSuccess: creditPerSuccess,
    }
}

func (b *RetryBudget) RecordSuccess() {
    b.mu.Lock()
    defer b.mu.Unlock()
    b.tokens += b.creditPerSuccess
    if b.tokens > b.capacity {
        b.tokens = b.capacity
    }
}

func (b *RetryBudget) TryTake() bool {
    b.mu.Lock()
    defer b.mu.Unlock()
    if b.tokens < 1 {
        return false
    }
    b.tokens--
    return true
}

The implementation deliberately keeps the budget small and predictable.

RecordSuccess adds credits without exceeding the configured capacity.

TryTake atomically checks and consumes a token. This prevents concurrent requests from spending the same token.

For example:

budget := NewRetryBudget(
    20,   // Maximum tokens
    0.1,  // Credit per successful initial request
)

A successful initial request adds 0.1 tokens. A retry requires one complete token.

The budget starts empty. This is intentional: a newly started instance does not immediately receive permission to generate retries. If a service needs an initial retry allowance, that can be configured explicitly.

5. Building a Retry Executor

A retry budget is only one part of a retry policy.

A useful retry executor should also support:

• A maximum number of attempts
• Exponential backoff
• Jitter
• Context cancellation
• Retryable error classification
• A shared retry budget

First, define the policy:

package retry

import (
    "context"
    "errors"
    "math/rand"
    "time"
)

var ErrInvalidPolicy = errors.New("invalid retry policy")

type Policy struct {
    MaxAttempts  int
    InitialDelay time.Duration
    MaxDelay     time.Duration
    Budget      *RetryBudget
    ShouldRetry func(error) bool
}

MaxAttempts includes the original attempt.

For example, MaxAttempts: 3 means one original attempt and at most two retries.

Next, calculate the exponential backoff.

func backoff(
    attempt int,
    initial time.Duration,
    maximum time.Duration,
) time.Duration {
    delay := initial

    for i := 1; i < attempt; i++ {
        if delay >= maximum/2 {
            return maximum
        }
        delay *= 2
    }

    if delay > maximum {
        return maximum
    }

    return delay
}

The delay grows exponentially until it reaches the configured maximum.

For example:

RetryMaximum delay1100ms2200ms3400ms4800ms51.6s

The next step is jitter.

Without jitter, clients that fail at approximately the same time may retry at approximately the same time. This creates synchronized traffic spikes.

Full jitter chooses a random delay between zero and the calculated maximum:

func fullJitter(maxDelay time.Duration) time.Duration {
    if maxDelay <= 0 {
        return 0
    }

    return time.Duration(
        rand.Float64() * float64(maxDelay),
    )
}

The package-level math/rand functions are safe for concurrent use. If you need reproducible tests, inject a random source rather than depending on global randomness.

Now we can implement the retry executor.

func Do(
    ctx context.Context,
    policy Policy,
    operation func(context.Context) error,
) error {
    if policy.MaxAttempts < 1 ||
        policy.InitialDelay <= 0 ||
        policy.MaxDelay < policy.InitialDelay ||
        operation == nil {
        return ErrInvalidPolicy
    }

    if policy.ShouldRetry == nil {
        return ErrInvalidPolicy
    }
    for attempt := 1; attempt <= policy.MaxAttempts; attempt++ {
        if err := ctx.Err(); err != nil {
            return err
        }
        err := operation(ctx)
        if err == nil {
            // Only successful initial attempts replenish the budget.
            if attempt == 1 && policy.Budget != nil {
                policy.Budget.RecordSuccess()
            }
            return nil
        }
        if attempt == policy.MaxAttempts {
            return err
        }
        if !policy.ShouldRetry(err) {
            return err
        }
        // Do not generate more traffic when the shared
        // retry budget has been exhausted.
        if policy.Budget != nil && !policy.Budget.TryTake() {
            return err
        }
        maxDelay := backoff(
            attempt,
            policy.InitialDelay,
            policy.MaxDelay,
        )
        delay := fullJitter(maxDelay)
        timer := time.NewTimer(delay)
        select {
        case <-ctx.Done():
            if !timer.Stop() {
                <-timer.C
            }
            return ctx.Err()
        case <-timer.C:
        }
    }
    return nil
}

A few design decisions are worth highlighting.

First, the budget is consumed before the retry.

Once the service decides to schedule another attempt, it spends a token. If the context is canceled during the backoff period, that token is not refunded.

This is a conservative choice. The budget controls permission to generate retry work, not only completed downstream calls.

Second, only successful initial attempts replenish the budget.

A successful retry does not replenish it. Otherwise, a service could keep generating retries and effectively fund its own retry traffic during a degraded period.

Third, the context is checked before each attempt and during backoff.

A canceled request should not continue generating work.

The executor returns the last operation error when the retry budget is exhausted. This preserves the underlying failure for callers, although production implementations may also expose a separate retry-denied metric or callback.

6. Choosing Which Errors Are Retryable

Not every failure should trigger a retry.

A retry policy should distinguish between transient failures and permanent failures.

Examples of potentially retryable failures:

• Temporary network errors
• Connection resets
• Selected HTTP 5xx responses
• HTTP 429 responses, subject to server guidance
• Temporary resource unavailability

Examples of failures that generally should not be retried:

• Invalid input
• Authentication or authorization failures
• Unsupported operations
• Validation errors
• Requests that have already exceeded their deadline

A simple classifier might look like this:

func shouldRetry(err error) bool {
    var netErr net.Error

    if errors.As(err, &netErr) {
        return netErr.Timeout() || netErr.Temporary()
    }
    return false
}

This is only an illustration. net.Error.Temporary() is deprecated, and real applications should classify errors according to their HTTP client, database driver, RPC framework, and dependency-specific semantics.

For HTTP clients, for example, a 429 response may include Retry-After. A robust policy should respect that server-provided delay where applicable, while still enforcing the request deadline and retry budget.

Also remember that retrying a request is not always safe.

A GET request is generally easier to retry than a payment or order-creation request. For operations with side effects, use idempotency keys or another deduplication mechanism.

A retry budget controls traffic. It does not make an operation safe to execute twice.

7. Putting It Together

Let’s use the retry executor with a market data provider.

budget := retry.NewRetryBudget(
    20,
    0.1,
)

policy := retry.Policy{
    MaxAttempts:  3,
    InitialDelay: 100 * time.Millisecond,
    MaxDelay:     2 * time.Second,
    Budget:       budget,
    ShouldRetry:  shouldRetry,
}

ctx, cancel := context.WithTimeout(
    context.Background(),
    3*time.Second,
)

defer cancel()

err := retry.Do(
    ctx,
    policy,
    func(ctx context.Context) error {
        return marketDataClient.Fetch(ctx, "AAPL")
    },
)

if err != nil {
    // Handle the final operation error.
}

The behavior is now bounded by multiple independent controls:

• The request can make at most three attempts.
• The request cannot exceed its context deadline.
• Each retry uses exponential backoff with jitter.
• The service must have an available retry token.
• Only successful initial attempts replenish the shared budget.

This is much safer than allowing every request to retry independently.

8. Retry Budgets and Other Resilience Patterns

Retry budgets do not replace other resilience mechanisms. They complement them.

MechanismPrimary responsibilityTimeoutLimits how long an operation can runBackpressureControls how much work enters a systemLoad sheddingRejects work when capacity is insufficientBulkheadIsolates resource consumptionCircuit breakerStops calls to an unhealthy dependencyRetry budgetLimits additional traffic generated by retries

Consider how these mechanisms work together.

A request enters the service. Backpressure controls admission. A bulkhead limits concurrent calls to the provider. The request has a deadline. If the provider fails, the retry policy classifies the error and checks the retry budget. If the dependency is persistently unhealthy, the circuit breaker can stop further calls.

Each mechanism addresses a different failure mode.

A subtle interaction: Bulkheads and retries

Retry placement matters.

Suppose a bulkhead permit is acquired once around the entire retry loop. The permit remains occupied during backoff.

That may be desirable if the goal is to limit the number of complete operations, but it can also waste scarce concurrency capacity while requests are sleeping.

Alternatively, acquiring a permit for each individual attempt releases capacity during backoff. However, this allows more logical requests to remain in progress simultaneously.

Neither approach is universally correct. The choice depends on what the bulkhead is protecting:

• If it protects active downstream calls, acquire a permit per attempt.
• If it protects the number of in-flight logical operations, hold the permit across the complete retry operation.

The important thing is to define the resource boundary explicitly.

9. Is a Local Retry Budget Enough?

A mutex-protected retry budget is local to one process.

If you run ten replicas, each replica has its own budget.

For example:

• 10 replicas
• 20 tokens per replica
• 200 tokens of aggregate retry capacity

This may be perfectly acceptable. Local budgets are simple, fast, and do not introduce another distributed dependency.

But if the external provider enforces a strict global quota, local budgets alone may not be sufficient.

Possible approaches include:

1. Local budgets: Simple and independent, but aggregate capacity grows with replica count.
2. Centralized budget: Coordinates retry allowance across instances, but introduces network calls and another failure boundary.
3. Provider-aware limits: Use the provider’s documented quota and response headers to coordinate traffic.
4. Centralized dispatch: Route downstream work through a shared queue or dispatcher that controls the total attempt rate.

Do not introduce a distributed retry-budget service automatically. The coordination mechanism can become more expensive and less reliable than the problem it is trying to solve.

Start with local budgets when they provide sufficient protection. Move to centralized coordination only when the actual quota or failure model requires it.

10. Observability: Measure Retry Behavior

A retry policy that cannot be observed is difficult to operate.

At minimum, expose metrics that help answer:

• How often are requests retried?
• How many retries are denied by the budget?
• Which dependencies consume the most retry capacity?
• How often do requests exhaust their attempt limit?
• How much time is spent waiting between attempts?

Useful metrics include:

MetricTypePurposeretry_attempts_totalCounterTotal retry attemptsretry_budget_denied_totalCounterRetries rejected by the budgetretry_budget_tokensGaugeCurrent available tokensretry_exhausted_totalCounterRequests reaching the attempt limitretry_delay_secondsHistogramTime spent in retry backoff

Avoid using raw URLs, request IDs, user IDs, or other high-cardinality values as metric labels.

Prefer bounded dimensions such as dependency name, operation, and outcome.

A particularly useful signal is the ratio of retries to original requests.

If the retry rate starts increasing while the initial success rate drops, the system may be entering a retry-amplification cycle.

The retry budget should make that cycle bounded, but the metrics help you understand why it happened.

11. Testing Retry Budgets

Concurrency behavior should be tested explicitly.

The first test verifies that successful initial requests replenish the budget and that retries consume tokens.

func TestRetryBudget(t *testing.T) {
    budget := NewRetryBudget(2, 1)

    budget.RecordSuccess()
    budget.RecordSuccess()
    if !budget.TryTake() {
        t.Fatal("expected first token")
    }
    if !budget.TryTake() {
        t.Fatal("expected second token")
    }
    if budget.TryTake() {
        t.Fatal("expected budget to be exhausted")
    }
}

The next test checks that concurrent callers cannot consume more tokens than are available.

func TestRetryBudgetConcurrent(t *testing.T) {
    const (
        tokens   = 100
        workers  = 1000
    )

   budget := NewRetryBudget(tokens, 1)
    for i := 0; i < tokens; i++ {
        budget.RecordSuccess()
    }
    var (
        wg       sync.WaitGroup
        accepted atomic.Int64
    )
    wg.Add(workers)
    for i := 0; i < workers; i++ {
        go func() {
            defer wg.Done()
            if budget.TryTake() {
                accepted.Add(1)
            }
        }()
    }
    wg.Wait()
    if got := accepted.Load(); got != tokens {
        t.Fatalf(
            "expected %d successful acquisitions, got %d",
            tokens,
            got,
        )
    }
}

Run the tests with the race detector:

go test -race ./...

You should also test the retry executor itself:

• A successful first attempt should not retry.
• A transient failure should retry when budget is available.
• A permanent failure should return immediately.
• An exhausted budget should prevent another attempt.
• Context cancellation should interrupt backoff.
• The maximum attempt count should be respected.
• A successful retry should not replenish the budget.
• A failed initial request should not replenish the budget.

Finally, test behavior under load. A retry policy that passes unit tests can still create unexpected traffic patterns when thousands of requests fail together.

12. Production Checklist

Before enabling retries in a production service, verify the following:

• [ ] Maximum attempts include the original request.
• [ ] Retryable errors are explicitly classified.
• [ ] Every request has a deadline or timeout.
• [ ] Backoff is exponential and bounded.
• [ ] Jitter prevents synchronized retries.
• [ ] A shared budget limits aggregate retry traffic.
• [ ] Non-idempotent operations have appropriate protection.
• [ ] Provider-specific rate limits and Retry-After are respected.
• [ ] Retry behavior is observable.
• [ ] Bulkhead permit ownership is clearly defined.
• [ ] Tests cover cancellation, concurrency, and exhaustion.
• [ ] Replica count and aggregate retry capacity are understood.

Conclusion

Retries are useful, but unlimited retry behavior can turn a temporary dependency failure into a system-wide incident.

A maximum attempt count protects individual requests. A retry budget protects the service from the combined effect of those requests.

The key idea is simple: successful traffic can fund a limited amount of additional work, but retries should not be allowed to generate unlimited traffic when the system is already unhealthy.

In Go, a concurrency-safe token bucket provides a straightforward way to implement this policy. Combined with deadlines, exponential backoff, jitter, bulkheads, and circuit breakers, retry budgets help make failure behavior more predictable.

Resilience is not about trying again forever. It’s about knowing when another attempt is worth the cost.

What ACP is, how it relates to MCP, how to integrate it into a TypeScript application, and what it means for teams building agent-powered developer tools.

1. What Is ACP, and What Problem Does It Solve?

When building AI-powered applications, the usual approach is to connect to a model provider, send a prompt, process the response, and optionally handle tool calls.

Coding agents introduce a different set of requirements.

Tools such as Claude Code, Gemini CLI, and Codex CLI can work with a codebase, inspect files, execute commands, make changes, and perform multi-step development tasks. Integrating these capabilities into our own applications requires more than simply displaying a model’s text response.

Without a common protocol, each client may need a custom integration for every agent it wants to support. Agents may also need to implement different interfaces for different clients.

The Agent Client Protocol (ACP) addresses this problem by standardizing communication between coding agents and client applications. It was designed primarily for coding agents and code editors, including IDEs, but its architecture can also support other agent-enabled client interfaces.

ACP defines how a client and an agent communicate, establish sessions, exchange messages, report progress, and handle interactions such as permission requests.

The key point is that ACP is not an LLM or a model provider. It is a communication protocol for integrating agents into applications.

This distinction matters because a coding agent is more than a text-generation endpoint. It may have its own execution loop, tools, filesystem access, authentication requirements, and session lifecycle.

Reference: Agent Client Protocol — Introduction

2. ACP vs. MCP: What Is the Difference?

ACP and the Model Context Protocol (MCP) are complementary, not competing, technologies. They address different parts of an agent-based architecture.

• ACP (Agent Client Protocol) standardizes communication between an agent and its client.
• MCP (Model Context Protocol) standardizes how AI applications and agents interact with tools and external context sources.

Consider a developer platform that allows an agent to review a GitLab merge request, inspect the related code, make a correction, and run tests.

The responsibilities could be divided as follows:

1. ACP connects the application to the coding agent and carries the interaction.
2. MCP makes the required tools available to the agent, such as GitLab operations or access to other services.
3. The Vercel AI SDK provides application-level APIs for consuming agent output, including text generation and streaming through a familiar TypeScript interface.

A simplified architecture looks like this:

User
 |
 v
AI Application
 |
 v
Vercel AI SDK
 |
 v
ACP Provider
 |
 v
ACP-Compatible Agent
 |
 +----> Filesystem
 +----> Terminal / Tests
 +----> MCP Servers
           |
           +----> GitLab
           +----> Internal APIs
           +----> Other Services

This diagram illustrates a possible architecture, not a guarantee that every agent supports every capability shown. The available features depend on the agent, its ACP implementation, session configuration, and execution environment.

One important implementation detail is that the agent can connect to configured MCP servers directly. The application does not necessarily proxy every MCP request through itself.

This separation helps us design the system around clear responsibilities: ACP handles agent communication, MCP exposes tools, and the application owns the user experience and its policies.

References: ACP Architecture · MCP Documentation

3. What Does the ACP Provider in the Vercel AI SDK Offer?

The Vercel AI SDK has a community provider that integrates ACP-compatible agents through the LanguageModel interface.

The package is @mcpc-tech/acp-ai-provider.

It allows a TypeScript application to use an ACP-compatible agent through familiar AI SDK functions rather than implementing the underlying protocol integration from scratch.

Its documented capabilities include:

• Agent integration: Connect to supported agents such as Gemini CLI, Claude Code, and Codex CLI.
• Text generation: Use generateText to send a task and receive a result.
• Streaming: Consume output through streamText.
• MCP configuration: Configure MCP servers for the agent’s session.
• Tool-call reporting: Expose provider-defined tool-call information through the AI SDK interface.
• Process management: Start agent processes and manage their lifecycle through the provider.

The important architectural nuance is that exposing an agent through the LanguageModel interface does not turn that agent into a conventional LLM API.

The provider offers a common application interface, but the underlying system may still execute multi-step workflows, use external tools, and manage its own state.

This abstraction is useful, but it does not make all agents behave identically.

Reference: AI SDK — ACP Community Provider

4. How to Integrate ACP

Step 1: Install the dependencies

For a Node.js and TypeScript project, install the AI SDK and the ACP provider:

pnpm add ai @mcpc-tech/acp-ai-provider

Installing the package is only the first step. The selected ACP agent must also be installed, configured, and authenticated according to its own requirements.

Step 2: Create an ACP provider

The following example uses Gemini CLI with its documented ACP command-line option.

import { createACPProvider } from '@mcpc-tech/acp-ai-provider';
import { generateText } from 'ai';

const provider = createACPProvider({
  command: 'gemini',
  args: ['--experimental-acp'],
  session: {
    cwd: process.cwd(),
    mcpServers: [],
  },
});

const { text } = await generateText({
  model: provider.languageModel(),
  prompt: 'Analyze this codebase and identify its main components.',
});

console.log(text);

Let’s break down the configuration:

• command specifies the executable used to start the agent.
• args passes command-line arguments to that executable.
• cwd defines the working directory for the agent.
• mcpServers configures the MCP servers available to the session.
• provider.languageModel() exposes the configured agent through the AI SDK's model interface.

For example, setting cwd to a repository directory allows the agent to work in that context, subject to its capabilities and permissions.

The command-line arguments are agent-specific. The example assumes a compatible Gemini CLI version is installed and supports the documented ACP mode.

Step 3: Stream the response

For a long-running coding task, waiting for the entire response before displaying anything can produce a poor user experience.

The AI SDK provides streamText to consume output incrementally:

import { createACPProvider } from '@mcpc-tech/acp-ai-provider';
import { streamText } from 'ai';

const provider = createACPProvider({
  command: 'claude-code-acp',
  session: {
    cwd: process.cwd(),
    mcpServers: [],
  },
});

const { textStream } = streamText({
  model: provider.languageModel(),
  prompt: 'Review the error handling in this repository.',
});

for await (const chunk of textStream) {
  process.stdout.write(chunk);
}

This pattern lets the application consume text as it becomes available.

However, text streaming and agent execution events are not the same thing. A coding interface may also need to display tool calls, file changes, permission requests, and execution progress.

Before building that experience, inspect the provider’s supported event mapping and determine which events are exposed through the version you use.

Step 4: Provide MCP servers to the agent

ACP does not automatically grant an agent access to GitLab, a database, or another internal service. Those capabilities need to be configured and made available to the agent.

For example, the following configuration provides a filesystem MCP server restricted to /tmp:

const provider = createACPProvider({
  command: 'gemini',
  args: ['--experimental-acp'],
  session: {
    cwd: process.cwd(),
    mcpServers: [
      {
        type: 'stdio',
        name: 'filesystem',
        command: 'npx',
        args: [
          '-y',
          '@modelcontextprotocol/server-filesystem',
          '/tmp',
        ],
      },
    ],
  },
});

This follows the provider documentation’s MCP configuration pattern. Verify the selected MCP server’s current package, installation instructions, and compatibility before using it.

A few details are worth keeping in mind:

1. The agent must support the MCP configuration and transport being used.
2. Filesystem and other tool permissions should be restricted to the minimum required.
3. Providing an MCP server does not replace authentication, authorization, or execution isolation.

There is also a distinction between configuring MCP tools for the agent and exposing tool calls through the AI SDK.

The provider documentation shows passing provider.tools to AI SDK functions when tool-call information should be available through the SDK interface:

const result = await generateText({
  model: provider.languageModel(),
  prompt: 'List the files in the configured workspace.',
  tools: provider.tools,
});

Here, provider.tools refers to the provider-defined tools. It is not a replacement for configuring the MCP servers that supply capabilities to the agent.

This distinction becomes important when designing tool-call interfaces, logging agent activity, or building an interactive coding environment.

Reference: AI SDK — ACP Provider: Examples and Tool Usage

5. Advantages of Using ACP

Reuse existing coding agents

Building a capable coding agent from scratch involves much more than prompt engineering. It may require repository context management, file operations, command execution, tool orchestration, and recovery from failures.

Integrating an existing ACP-compatible agent can reduce the amount of infrastructure we need to implement ourselves.

This is especially useful when we want to experiment with coding workflows without immediately committing to maintaining our own full agent runtime.

Reduce client-agent integration overhead

ACP introduces a common communication interface for compatible clients and agents.

Instead of writing a completely different integration for every combination, an application can build against the protocol and add support for compatible agents.

There will still be agent-specific configuration and behavioral differences, but the common protocol can reduce duplicated integration work.

Build our own product experience

We do not necessarily have to expose developers to the agent’s original terminal or editor interface.

We can build a custom interface in a web application, desktop client, or internal developer platform.

Possible features include:

• Task creation and tracking
• Streaming output and progress indicators
• Repository and session selection
• Permission-request dialogs
• Tool-call visibility
• File-change summaries
• Integration with existing code review workflows

These are application features built on top of the protocol, not turnkey features supplied by ACP itself.

Reuse MCP-based capabilities

Organizations that already maintain MCP servers may be able to reuse those integrations with compatible agents.

For example, an agent could use one set of configured tools to retrieve a merge request, another to read internal documentation, and another to run approved operations.

The benefit is architectural reuse: tools can be maintained as a separate capability layer rather than being embedded into every agent integration.

6. Limitations and Trade-offs

ACP simplifies communication, but it does not eliminate the complexity of operating coding agents.

The provider is community-maintained

The AI SDK ACP integration is a community provider, not a first-party Vercel model provider.

Being listed in the AI SDK documentation does not mean it has the same ownership, support commitments, or release lifecycle as a first-party package.

Before adopting it in production, evaluate maintenance activity, release compatibility, issue response, and the cost of maintaining a fallback or internal adapter.

Model selection is currently limited

The provider documentation states that it exposes a single language model representing the configured agent and does not currently support selecting a specific underlying model through its model interface.

This matters in environments where model choice is a platform-level concern.

For example, an organization might require specific models for latency, cost, data-handling, or compliance reasons. If model selection is controlled by the agent’s own configuration, that control may not be available through the ACP provider itself.

The current limitations should be checked again before adoption because community provider features can change.

Process lifecycle and concurrency need attention

The documented provider runs in Node.js environments with child-process capabilities. Its documentation also notes that each language model instance spawns a new agent process.

That creates operational questions:

• When are agent processes started and terminated?
• How are timeouts and cancellations handled?
• What happens if an agent exits unexpectedly?
• Can sessions be resumed, and under what conditions?
• How are concurrent tasks isolated?
• What limits apply to CPU, memory, and execution time?

Creating a new agent process for every request without a lifecycle strategy can lead to unnecessary startup overhead, resource consumption, and operational instability.

For a production application, process management should be part of the design rather than an implementation detail left to chance.

ACP is not a sandbox

An agent may read files, execute commands, and invoke tools. ACP standardizes the communication involved, but the protocol alone does not isolate those operations.

The effective security boundary depends on the execution environment, the agent, its tool configuration, and the permissions granted to it.

A production implementation should consider:

• Isolated workspaces or containers
• Least-privilege filesystem and terminal access
• Explicit approval for sensitive operations
• Secure credential handling
• Auditable command and file-change records
• Resource limits and execution timeouts
• Cancellation and recovery behavior

The client should not assume that every agent implements the same permission model or supports the same capabilities.

Compatibility does not mean identical behavior

Two ACP-compatible agents can differ in supported features, authentication methods, tools, session handling, and their interpretation of a task.

ACP reduces communication-level coupling, but it does not guarantee identical results when switching agents.

Applications should define a common baseline of supported capabilities and make agent-specific differences explicit.

References: AI SDK — ACP Provider Limitations · ACP Protocol Overview · ACP Security Considerations

7. What Can an AI Chapter Lead Contribute?

The most valuable outcome is not simply introducing another protocol. It is using the protocol to establish a repeatable approach to agent integration, security, and developer experience across teams.

1. Establish a shared agent integration layer

Without a common approach, teams may implement their own agent lifecycle management, streaming, permission handling, and error recovery.

A shared integration layer can provide common abstractions for:

• Agent configuration and selection
• Session creation and cleanup
• Output streaming
• Cancellation and timeouts
• Permission handling
• Error reporting and retries
• Logging and observability

This allows feature teams to focus on their workflows instead of rebuilding the same integration infrastructure.

The abstraction should not hide meaningful differences between agents. It should provide a common baseline while allowing explicit capability checks and agent-specific configuration.

2. Define when to use ACP, MCP, or a model provider

A short architecture guide can help teams decide which technology belongs at which layer.

RequirementPrimary considerationGenerate text or structured output from a modelAI SDK model providerGive an agent access to internal APIs or toolsMCPConnect a coding agent to a client applicationACPStream task output and supported progress eventsACP integration and AI SDKEnforce permissions and restrict executionApplication policies and runtime isolation

These are architectural guidelines, not mutually exclusive choices. A single system may use all three.

The objective is to avoid introducing a protocol simply because it is available. Every component should have a clear responsibility.

3. Define a secure execution standard

Coding agents introduce security concerns beyond those of a read-only chat interface.

Before teams use agents to make changes or run commands, establish standards for workspace isolation, tool permissions, credential management, user approvals, and auditability.

For instance, an agent that reviews a repository may only need read access. An agent that fixes a bug may require write access and permission to run tests. An agent that interacts with a shared environment should have a much stricter policy.

Permission decisions should be enforced by trusted application and runtime components rather than relying solely on an agent’s instructions.

4. Run a measurable pilot

A working demo shows that an integration is possible. It does not show that the integration is useful or production-ready.

A good pilot would be a bounded merge request review and correction workflow:

1. The agent inspects a repository and identifies potential issues.
2. It explains the relevant files and findings.
3. It proposes a change or modifies permitted files.
4. It runs the available tests.
5. The application presents a summary of the changes and test results.
6. A developer reviews the result before it is merged.

The pilot should be compared with the existing workflow using the same tasks wherever possible.

Useful metrics include task completion rate, end-to-end duration, process startup overhead, tool-call failures, resource consumption, human intervention, and the number of changes accepted after review.

Security checks and regression tests should be treated as acceptance criteria, not optional improvements.

5. Create a reference implementation and share learnings

Instead of asking each team to interpret protocol documentation independently, provide a small reference application.

It should demonstrate provider configuration, MCP integration, streaming, process lifecycle management, permission handling, and failure scenarios.

Teams can then reuse the implementation and contribute improvements.

This approach makes the learning transferable: the organization develops an integration standard and a practical understanding of where the technology helps, where it introduces complexity, and which limitations still need work.

8. Questions to Answer Before Production Adoption

Before choosing ACP for a production workflow, I would evaluate the following:

• Does our selected agent implement ACP, and which version and capabilities does it support?
• Does the provider expose the events and controls required by our application?
• Where does the agent process run, and what resources can it access?
• How are credentials, permissions, and approval decisions managed?
• How are sessions, cancellation, timeouts, and process termination handled?
• How are concurrent users and tasks isolated?
• How will we measure reliability, cost, performance, and developer productivity?
• What is our plan if the community provider becomes incompatible with a new AI SDK release?

The key is to separate protocol compatibility from operational readiness.

A compatible agent may be easy to connect but still require substantial engineering around security, process management, observability, and user experience.

Conclusion

ACP is worth evaluating when we want to integrate coding agents into our own applications without building every agent-specific communication layer from scratch.

The Vercel AI SDK’s community provider makes this approach accessible to TypeScript applications by exposing ACP-compatible agents through familiar APIs.

But ACP should not be treated as a universal replacement for model APIs or as an alternative to MCP. Its value is more specific: it standardizes communication between an agent and the client that uses it, while MCP can provide the tools an agent needs.

For an AI Chapter Lead, the opportunity is to turn that capability into a reusable engineering practice: define clear architectural boundaries, standardize secure integration patterns, run controlled pilots, and evaluate outcomes using measurable criteria.

My recommended starting point: choose one bounded coding workflow, build a small ACP-based reference implementation, and compare it with the existing approach. Make the production decision based on compatibility, operational effort, security, and measured results — not simply on whether the demo works.

References and Further Reading

1. AI SDK — ACP Community Provider — Installation, configuration, examples, tool usage, and documented limitations.
2. Agent Client Protocol — Introduction — ACP’s purpose and the interoperability problem it addresses.
3. Agent Client Protocol — Architecture — Client-agent communication and MCP integration.
4. Agent Client Protocol — Protocol Overview — Sessions, messages, progress updates, and permission requests.
5. Agent Client Protocol — Security Considerations — Security boundaries and integration considerations.
6. Agent Client Protocol — TypeScript SDK — The TypeScript SDK for building ACP clients and agents.
7. Model Context Protocol Documentation — MCP concepts, tools, and integration patterns.
8. MCPC — GitHub Repository — The project associated with the ACP AI SDK provider.

A service can have healthy code, reasonable timeouts, and a circuit breaker — and still suffer a cascading failure.

Imagine a Go API that serves portfolio data, market prices, and investment reports. It calls three external dependencies:

• A market data provider
• A portfolio database
• A reporting service

The reporting service becomes slow. Requests begin waiting for it, goroutines remain occupied, and incoming traffic continues to grow. Eventually, requests that do not even need reports start failing because they compete for the same resources.

The original failure was isolated to one dependency. The application’s resource usage was not.

This is the problem the bulkhead pattern is designed to address.

In this article, we’ll build dependency-specific bulkheads in Go, explore admission and waiting policies, examine common implementation mistakes, and discuss how to verify that isolation actually works under load.

1. What is the bulkhead pattern?

The name comes from the watertight compartments inside a ship. If one compartment floods, the others should remain intact.

In software, a bulkhead creates a resource boundary around a particular dependency, workload, or tenant. When one boundary reaches capacity, work inside that boundary is limited without automatically consuming capacity reserved for others.

Consider a service with three downstream dependencies.

Without isolation:

┌────────────────────┐
Incoming ───────►│   Shared Capacity  │
                 └─────────┬──────────┘
                           │
                ┌──────────┼──────────┐
                ▼          ▼          ▼
             Market     Portfolio   Reporting
             Data       Database    Service

If reporting requests become slow, they can consume the shared capacity needed by market data and portfolio requests.

With dependency-specific bulkheads:

┌────────────────────┐
Incoming ───────►│    Go API          │
                 └─────────┬──────────┘
                           │
              ┌────────────┼────────────┐
              ▼            ▼            ▼
          ┌────────┐   ┌────────┐   ┌────────┐
          │ Market │   │Portfolio│  │Report  │
          │ Limit  │   │ Limit   │  │ Limit  │
          └───┬────┘   └───┬────┘   └───┬────┘
              ▼            ▼            ▼
          Market API    Database     Report API

Each dependency receives its own concurrency budget.

If the reporting bulkhead is full, new reporting work can be rejected or briefly delayed according to policy. Market data and portfolio operations retain their own permits.

The important property is not that failures disappear. It is that one dependency cannot consume all the capacity reserved for unrelated work.

2. Bulkheads are not circuit breakers

Both patterns help prevent cascading failures, but they protect against different problems.

PatternWhat it protects againstTimeoutWaiting too long for an operationRate limiterExcessive request frequencyLoad sheddingAccepting more work than the service can safely handleCircuit breakerRepeated calls to a dependency that is failingBulkheadOne workload consuming resources needed by another

Suppose a reporting API has a 10-second response time.

A timeout limits how long a caller waits. A circuit breaker may stop future calls after repeated failures. Neither automatically prevents 100 concurrent reporting calls from consuming every available application slot.

A bulkhead limits that concurrency independently.

These patterns complement one another. A typical dependency call may use a bulkhead, a deadline, and a circuit breaker, with a carefully bounded retry policy where retries are safe.

3. Start with a bounded semaphore

Go’s channel primitives provide a straightforward way to limit concurrent operations.

A buffered channel can represent available permits. Each operation acquires a permit before executing and releases it when finished.

For a dependency that should allow at most 10 concurrent calls:

var reportingSlots = make(chan struct{}, 10)

However, the acquisition policy matters.

If all permits are occupied, should a new call wait for capacity or fail immediately?

For an already-slow dependency, waiting indefinitely can simply move the queue into application memory. In many latency-sensitive paths, a non-blocking bulkhead is preferable.

Let’s implement a reusable version.

4. Implementing a fail-fast bulkhead

The following implementation limits active operations and returns a recognizable error when capacity is exhausted.

package bulkhead

import (
    "context"
    "errors"
)

var ErrFull = errors.New("bulkhead capacity exhausted")

type Bulkhead struct {
    slots chan struct{}
}

func New(maxConcurrent int) *Bulkhead {
    if maxConcurrent <= 0 {
        panic("maxConcurrent must be greater than zero")
    }
    return &Bulkhead{
        slots: make(chan struct{}, maxConcurrent),
    }
}

func (b *Bulkhead) Do(
    ctx context.Context,
    fn func(context.Context) error,
) error {
    if err := ctx.Err(); err != nil {
        return err
    }
    select {
    case b.slots <- struct{}{}:
        // A permit has been acquired.
    case <-ctx.Done():
        return ctx.Err()
    default:
        return ErrFull
    }
    defer func() {
        <-b.slots
    }()
    // Cancellation may race with permit acquisition.
    if err := ctx.Err(); err != nil {
        return err
    }
    return fn(ctx)
}

This implementation deliberately does not queue callers waiting for a permit.

If capacity is available, the operation proceeds. If every slot is occupied, the caller receives ErrFull immediately.

The second context check handles the case where cancellation races with permit acquisition. If the context is canceled by that point, the deferred release returns the permit.

The function passed to Do should honor its context and should not start detached work that outlives the operation without an independent lifecycle policy.

Using the bulkhead

var reportBulkhead = bulkhead.New(10)

func GenerateReport(
    ctx context.Context,
    reporter Reporter,
) error {
    return reportBulkhead.Do(ctx, func(
        ctx context.Context,
    ) error {
        return reporter.Generate(ctx)
    })
}

Reporter is an application-defined interface in this example.

The reporting dependency now has an independent concurrency limit. If it becomes slow and all 10 permits are occupied, additional reporting calls fail fast rather than accumulating behind the dependency.

The limit is a policy choice, not a universal recommendation. Choose it based on load testing, downstream capacity, latency objectives, and the number of application instances.

5. Choose the right rejection behavior

A full bulkhead does not always need to produce an HTTP error directly. The appropriate response depends on the caller and the operation.

For example:

func handleReport(
    w http.ResponseWriter,
    r *http.Request,
) {
    err := GenerateReport(r.Context(), reporter)

    switch {
    case errors.Is(err, bulkhead.ErrFull):
        http.Error(
            w,
            "reporting capacity temporarily exhausted",
            http.StatusServiceUnavailable,
        )
        return
    case errors.Is(err, context.Canceled):
        // The client or upstream caller canceled the request.
        return
    case errors.Is(err, context.DeadlineExceeded):
        http.Error(
            w,
            "report generation timed out",
            http.StatusGatewayTimeout,
        )
        return
    case err != nil:
        http.Error(
            w,
            "report generation failed",
            http.StatusBadGateway,
        )
        return
    }
    w.WriteHeader(http.StatusNoContent)
}

This is a simplified handler; the application would need to define reporter, imports, and its actual response contract.

The important distinction is between admission rejection, caller cancellation, deadline expiration, and dependency failure. They are different events and should not be collapsed into one generic error.

For public APIs, document the overload response and decide whether a 503 or another status is appropriate for the contract. Avoid encouraging immediate retries when the dependency is already saturated.

6. A bulkhead must protect the actual bottleneck

A semaphore limits how many operations enter a code path. It does not automatically isolate every resource used by that path.

Consider a Go service that calls three downstream APIs through a single http.Client.

The client uses a shared http.Transport, which manages connection pooling and other transport resources. Even if the application has separate concurrency semaphores, shared transport configuration or other shared resources may still create contention.

A useful design starts by identifying what is being protected:

• Concurrent requests to a dependency
• Connections to a database
• Worker capacity for background jobs
• CPU-intensive computation
• Memory-intensive report generation
• Per-tenant capacity
• External provider quotas

Then enforce limits at the relevant resource boundary.

For outbound HTTP calls, use dedicated transports when separate connection-pool configuration or transport-level isolation is useful:

func newHTTPClient(maxConns int) *http.Client {
    transport := &http.Transport{
        MaxConnsPerHost:     maxConns,
        MaxIdleConns:        maxConns,
        MaxIdleConnsPerHost: maxConns,
    }

    return &http.Client{
        Transport: transport,
        Timeout:   2 * time.Second,
    }
}
var (
    marketClient = newHTTPClient(20)
    reportClient = newHTTPClient(5)
)

This snippet requires the net/http and time imports.

The clients have separate transports and different connection limits. The timeout is an example; production systems should choose deadlines based on their latency budgets and may prefer request contexts plus explicit transport settings.

Connection limits are not a replacement for application-level concurrency limits. A connection can be reused, and one operation can involve more than one resource. Use both controls when they protect distinct bottlenecks.

Also remember that a custom transport changes the default transport behavior. Configure TLS, proxy handling, connection lifetimes, idle timeouts, and other relevant settings according to your environment.

7. Waiting queues: useful, but not free

Fail-fast admission is not always the best policy.

Some operations can tolerate a short wait for capacity. In that case, a bounded waiting policy may improve utilization without allowing unbounded accumulation.

There are three common approaches:

1. Fail fast: Reject immediately when no permit is available.
2. Wait with a deadline: Wait for a permit, but only until the caller’s context expires.
3. Bound active work and queued work separately: Allow a defined number of operations to run and a small number to wait.

The third approach requires particular care. A semaphore that limits active calls does not limit the number of goroutines waiting to acquire it.

A context-aware waiting version can be implemented like this:

func (b *Bulkhead) DoWait(
    ctx context.Context,
    fn func(context.Context) error,
) error {
    if err := ctx.Err(); err != nil {
        return err
    }

    select {
    case b.slots <- struct{}{}:
    case <-ctx.Done():
        return ctx.Err()
    }
    defer func() {
        <-b.slots
    }()
    if err := ctx.Err(); err != nil {
        return err
    }
    return fn(ctx)
}

This method waits for a permit or cancellation. It does not bound the number of callers waiting in goroutines.

If the service can receive an unbounded number of waiting callers, use an explicit bounded admission queue, reject before creating excess work, or use a bounded worker pool.

The distinction is important: a concurrency limit without a queue bound can still leave the system with too many waiting goroutines.

8. Isolate workloads, not just services

Dependency-level bulkheads are useful, but some applications need additional boundaries.

Consider a service with the following operations:

• Interactive portfolio queries
• Historical analytics
• Exporting large reports
• Background notifications

These workloads may share dependencies while having very different resource profiles.

A report export can occupy database connections for much longer than a small portfolio query. If both share the same database pool and the same admission policy, the reporting workload may starve interactive traffic.

Possible isolation strategies include:

• Separate concurrency limits by workload.
• Dedicated worker pools for long-running jobs.
• Bounded queues for background processing.
• Database connection budgets for distinct workloads, where operationally feasible.
• Reserved capacity for latency-sensitive operations.
• Per-tenant limits to prevent one customer from dominating capacity.

Do not create a separate pool for every function by default. Each additional boundary introduces configuration, monitoring, and capacity-planning costs.

The goal is meaningful isolation around resources that have different failure modes or service-level requirements.

9. Beware of multiplicative concurrency

A local bulkhead limit is not necessarily a global limit.

Suppose a service runs 8 replicas, and each replica allows 20 concurrent calls to a reporting provider.

The potential aggregate concurrency is:

[ 8 \times 20 = 160 ]

If the provider can safely handle only 50 concurrent calls, the local configuration does not protect the global dependency.

This is one of the most common capacity-planning mistakes in distributed systems.

When choosing limits, account for:

• The maximum number of replicas.
• Autoscaling behavior.
• Other applications calling the same dependency.
• Provider-wide quotas.
• Deployment overlap during rolling updates.
• Traffic distribution and uneven load.

A local bulkhead is excellent for protecting each process. A global quota may require a shared admission mechanism, provider-side limits, a centralized dispatcher, or a different partitioning strategy.

Avoid introducing a distributed semaphore unless global coordination is genuinely needed. The coordination service can become another dependency whose failure must be handled.

10. Combining bulkheads with circuit breakers and retries

These patterns solve different parts of the same failure scenario.

Imagine a reporting provider starts timing out.

1. The bulkhead limits how many reporting calls can be active.
2. The deadline prevents each call from waiting indefinitely.
3. The circuit breaker can stop repeated calls after a failure threshold is reached.
4. A retry policy may retry selected transient failures, within a strict budget.
5. Load shedding rejects additional work when admission capacity is exhausted.

Order and interaction matter.

If retries happen inside a bulkhead permit, the permit remains occupied across attempts and backoff periods. This can reduce pressure on the downstream service, but it also ties up local capacity while no useful work is being performed.

If retries happen outside the bulkhead, every attempt must reacquire capacity, which may make admission more responsive but can complicate retry scheduling.

Neither choice is universally correct. Define whether the permit represents a whole logical operation or one outbound attempt, then test the behavior under slow failures.

Never retry a bulkhead rejection immediately by default. The rejection indicates that local capacity is exhausted; an immediate retry simply competes for the same capacity again.

For operations with side effects, retries also require idempotency or another mechanism that makes repeated attempts safe.

11. Observability: prove that isolation works

A bulkhead without metrics is difficult to tune and nearly impossible to validate during an incident.

For each bulkhead, consider recording:

• Active operations
• Configured concurrency capacity
• Rejected acquisitions
• Wait duration, if waiting is allowed
• Operation duration
• Timeouts and cancellations
• Downstream error rates
• Circuit-breaker state, where applicable

Useful metric names might include:

dependency_bulkhead_active
dependency_bulkhead_capacity
dependency_bulkhead_rejected_total
dependency_request_duration_seconds
dependency_request_errors_total

Use bounded labels such as a dependency name or workload class. Avoid raw user IDs, request IDs, and other high-cardinality values.

Monitor the bulkheads independently. A global saturation metric can hide the fact that reporting is exhausted while market data remains healthy.

A useful operational question is not merely, “Is the service saturated?”

It is: Which resource boundary is saturated, how long has it been saturated, and are unrelated workloads still meeting their latency objectives?

12. Test the failure boundary

A bulkhead should be tested with a deliberately slow dependency.

For example, configure a dependency bulkhead with a capacity of two, start two operations that block on a test-controlled channel, and then submit a third operation.

The third operation should receive ErrFull without invoking its callback.

A simplified test:

func TestBulkheadRejectsWhenFull(t *testing.T) {
    b := bulkhead.New(2)

    release := make(chan struct{})
    started := make(chan struct{}, 2)
    operation := func(context.Context) error {
        started <- struct{}{}
        <-release
        return nil
    }
    var wg sync.WaitGroup
    for range 2 {
        wg.Add(1)
        go func() {
            defer wg.Done()
            _ = b.Do(context.Background(), operation)
        }()
    }
    // Wait until both permits are occupied.
    <-started
    <-started
    err := b.Do(
        context.Background(),
        func(context.Context) error {
            t.Fatal("operation should not have been admitted")
            return nil
        },
    )
    if !errors.Is(err, bulkhead.ErrFull) {
        t.Fatalf("got %v, want %v", err, bulkhead.ErrFull)
    }
    close(release)
    wg.Wait()
}

This test requires the usual testing, context, sync, and errors imports, and the bulkhead package import. It assumes the test is in a separate package where the bulkhead package can be referenced.

The test uses synchronization channels rather than arbitrary sleeps to establish that both permits are occupied.

Additional tests should cover:

• Cancellation before acquisition.
• Cancellation racing with acquisition.
• Permit release after errors and panics.
• Recovery after a dependency becomes healthy.
• Waiting callers and bounded queues, if supported.
• Isolation between two independent bulkheads.
• Maximum aggregate concurrency across replicas.
• Saturation of one dependency while unrelated operations remain responsive.

For a full integration test, make one fake dependency slow or unresponsive and verify that requests to other dependencies still meet their expected latency and success-rate objectives.

13. Common mistakes

Using one shared limiter for every dependency. This can recreate the very contention the bulkhead was intended to prevent.

Bounding concurrency but not waiting callers. A semaphore can limit active operations while allowing many goroutines to accumulate while waiting.

Assuming a local limit is a global quota. Per-replica capacity multiplies across replicas and deployments.

Holding permits during unnecessary work. Long retry backoffs, unrelated processing, or detached tasks can occupy capacity longer than intended.

Creating too many isolated pools. Excessive fragmentation can leave some pools idle while others reject work. Isolation should follow real workload and resource boundaries.

Ignoring shared infrastructure. Separate application semaphores do not automatically isolate database pools, HTTP transports, CPU, memory, or downstream quotas.

Measuring only errors. A bulkhead may be working correctly by rejecting excess work. You need rejection, saturation, latency, and dependency health metrics to interpret the behavior.

14. Production checklist

Before deploying a bulkhead, verify that:

• The protected resource and isolation boundary are explicit.
• Each materially different dependency or workload has an appropriate capacity budget.
• Both active concurrency and queued work are bounded where necessary.
• Context deadlines and cancellation are propagated.
• Rejections are distinguishable from timeouts and dependency failures.
• Permits are released on every exit path.
• Retry policies do not amplify saturation.
• Per-instance limits are compatible with global dependency quotas.
• Metrics expose saturation and rejection per bulkhead.
• Tests demonstrate that one slow dependency cannot starve unrelated workloads.

Final thoughts

Bulkheads do not make dependencies reliable. They make dependency failures more contained.

In Go, channels, semaphores, dedicated clients, bounded worker pools, and resource-specific limits can establish useful isolation boundaries. The hard part is not writing a semaphore. It is deciding what must be isolated, how much capacity each workload deserves, and what should happen when that capacity is exhausted.

A reliable system should not allow one slow dependency to determine the fate of every other operation.

The goal is not to eliminate every failure. It is to keep one failure from becoming everyone else’s failure.