AI Agents (07): Agentic Harness (Frameworks)

From RAG, Orchestration, Observability and Tools to Skills — All You Need to Know About AI Agentic Harnesses & Frameworks

0s & 1s — All About Software ✨

12 min read

·

Sep 22, 2026

--

8

Listen

Share

Greetings from the author! If this is the first story you have come across in this series, don’t miss out on the entire AI Agents series — explaining in great detail the core of AI agents, their communication protocols, famous frameworks, agent scaling, monitoring and observability — and a lot more. We have a great multi-part Generative AI series as well; don’t forget to checkout that too!

💥 Master AI & Tech Skills
🎓 Get Up to 50% OFF Premium Courses
⏰ Limited-Time Offer
👉 Enroll Now & Start Learning

If you like my work and want to support it, buy me a drink on Ethereum or Bitcoin blockchain. All my 100+ articles on Medium are completely free and open for all.

Article Outline

Part-00: Introduction — Agentic Harness
— Available Frameworks (Harnesses)
— Main Layers of Agentic Frameworks

Part-01: The Orchestration Layer of the Agentic Harness
— Workflow Architectures
— Ahead-of-time Designed Workflows & Planned Stages
— Dynamic Workflow Design & Stage Planning
— Multi-agent Collaboration, Communication & Data Hand-offs (A2A)
— Human-in-the-Loop Approvals
— Checkpoints, Pausing, Resuming
— Automated State & Context Management
— Failure Recovery

Part-02: The Tools Layer of the Agentic Harness
— Internal Tools
— External Tools (API, MCP)

Part-03: The RAG Layer of the Agentic Harness
— Loading
— Parsing
— Chunking
— Embedding
— Storage
— Retrieval
— Re-ranking
— Ingestion

Part-04: The Observability Layer of the Agentic Harness
— Monitoring
— Tracing
— Logging
— Calculation
— Evaluation

Part-05: The Cloud Layer of the Agentic Harness
— Cloud Provider / Model Provider
— Vendor Lock-in Issues & Circumvention
— Ecosystem, Support & Tooling

This is going to be a rather long article. Taking time, and patiently reading through it is highly advised by the author.

Part-00: Introduction — Agentic Harness

Building scalable, secure, performant and dynamic AI agents from scratch is one of the most tedious modern software development tasks. Luckily, there are several frameworks out there which help us with automating the whole process by abstracting away the details.

— Agentic harness is the software-level infrastructure built around an LLM to equip it with the ability of taking action in the real world, while monitoring and logging everything it makes the agents do. —

A) Available Frameworks (Harnesses)

But there are so many frameworks out there. Pretty much every single big tech company has created at least one agentic development framework / kit. Take the following list as an example.

— Microsoft Semantic Kernel
— Microsoft Autogen
— Microsoft Agent Framework
— Google Agent Development Kit
— OpenAI Agents SDK
—LangChain & LangGraph
— CrewAI
— Mastra

…. and more. So, how do we decide which one to pick and which harness is the best for our developmental needs? Let me break that down next.

B) Main Layers of Agentic Frameworks

Well even though each particular framework specializes in some core set of values, nevertheless the first principles of building AI agents remain the same across all the frameworks.

In clearer words, all frameworks developing their agentic harnesses revolve around five main layers:

— Orchestration
— Tools
— RAG
— Observability
— Cloud & model ecosystem

If you learn the first principles of each of these layers to understand the agentic harness yourself, you cannot go wrong with picking a framework that you need for your organisational / project needs.

My goal, in this article, is to help you understand each of the afore-mentioned concepts, so that you can successfully, independently make decisions of your own when architecting the LLM/agent apps.

Part-01: The Orchestration Layer

— The Most Important Layer of an Agentic Harness; No Compromises Acceptable Here —

This is the most important layer of the framework. Orchestration in this context means the act of managing the end-to-end workflows, containing over a dozen nuances, which we will discuss shortly in this section. All AI agent frameworks come equipped with orchestration layers, which help us build graph-based, role-based, single- or multi-agent processes that operate autonomously or semi-autonomously.

1) Workflow Architectures

There are multiple techniques for orchestrating our workflows. Depending upon the nature of the workflow, you may want to go with one of the following or combine multiple to create a unique mix for your needs.

— Linear Vs. branched
— Sequential Vs. parallel
— Router (if-else)
— Centralized (hierarchical) Vs. Decentralized (non-hierarchical)

2) Ahead-of-time Workflow Design & Stage Planning

In order to bring more determinism, have more control, better maintainability and greater predictability / reproducability in our workflows, we may want to go with ahead-of-time pre-planned agentic workflows. Such workflows have already defined:

— Agents (names, descriptions, tasks, roles and end goals)
— Tasks (names, descriptions, expected output, I/O format)
— Tools
— Skills
— HITL checkpoints

… and more. This help us know very well ahead-of-time how the execution flow will move from the point of inception to the end.

3) Dynamic Workflow Design & Stage Planning

But sometimes it is just not possible for us to design workflows before execution, because the user’s prompt uniquely describes how to construct the workflow. Take this user prompt for example:

“Go to my Google Drive and get from there the file schedule.pdf. Read from there the schedule for today, including the meeting at 4 PM. Then put it in my calendar and also in the my to-do app. Finally, send the Email to Jacob for tonight’s fiesta. Make no mistakes.”

In this prompt, there is no way we can have a pre-defined workflow. So, an agentic framework MUST give the ability to break down tasks in real time, design the entire workflow, plan the stages, execute them in order, decide the architecture, verify the result and then ping the user via the interface — all via the help of the LLM and the available tools. All of this is super helpful for highly idiosyncratic workflows. In fact, dynamic workflows is the only choice we have for such use cases.

4) Multi-agent Collaboration & Communication

— Google’s Agent-to-Agent Communication Protocol —

When agents are designing the workflow and planning the stages, they need to divide tasks, delegate them, receive responses, coordinate all the data — and much more — to successfully execute the task at hand. For this, agents use what Google calls “Agent-to-Agnet” communication protocol (A2A). An agentic framework must provide collaboration and communication abstraction layer which is efficient, minimizes latency and doesn’t incur a lot CPU / GPU time.

The details of A2A can be found here, in this article, I wrote over 2 months ago.

5) HITL Approvals

— Human-in-the-Loop —

Often during agentic execution, we have to pause and get a human’s opinion. This can be done when calling a tool or after calling the tool or anywhere in between, whenever the engineers want to plan it.

Successfuly agentic orcestration requires good interfaces and abstractions for HITL approvals.

6) Checkpoints, Pausing & Resuming

In order for effective failure recovery, we need to checkpoint our execution state in some database / local file. Also, before HITL approval, we must:

• Save the context and the state of the LLM and the workflow in the DB
• Generate a unique UUID for the workflow state
• Generate a partial response for the front-end (user)
• Find the workflow in the DB
• Reinstate the workflow state and context
• Start executing again the whole thing from that point onwards
• Loop over the previous stage if HITL approval is not met

And that’s just for managing HITL approvals! This logic of managing workflow pausing, resuming and checkpointing is very complex and MUST be there in a framework that you want to choose for your organisational flows.

7) Failure Recovery

— Checkpionts & WALs —

Generating the checkpoints and logging them in a file on the server and storing them in the disk is absolutely cruicial for failure recovery. Once the application is reinstated on that server, the frameworks automatically starts executing the entire workflow again, causing no drama and no retries to be made from the UI. These logs are called WALs (Write-Ahead Logs).

8) State & Context Management

— “Memory” of the Workflow —

We must have the ability in the framework to successfully save somewhere in the DB what stages have been completed, what data was generated by them, etc. Also, at any given stage, the conversation happening with the LLM generates context. We need to store that context somewhere too. All of this helps with pausing/resuming the workflow after crashes as well as HITL checkpoints.

Frameworks usually call their state and context objects “memory”. LangGraph, for example, has a central state object which all the nodes in the graph refer to and update. The framework itself prevents race conditions in case of a parallel workflow, executing multiple stages in parallel or concurrently.

Part-02: The Tools Layer

— What Makes Agentic AI, “Agentic” —

If I were to really simplify the core definition of agentic AI, it would be captured by this equation.

Agentic AI = Generative AI (LLMs) + Tools + Auth

That’s it. That’s agentic AI for you in a nutshell.

Working in the roles of agentic AI architect and generative AI system designer, over the course of past several months, I have been able to simplify agentic AI to this one equation. And that’s what agentic AI really is: everything else is a fancy layer of abstraction, added as part of an agentic harness.

A) Internal Tools

File system crawlers, notification generators, internal DB access for reading, for example, are all internal tools. They are hosted / located in your own system, and you define authorization and access levels for these systems and assign connection strings with those authorization status to the agents.

Imagine you have PostgreSQL in your system. The agentic framework you are working with is CrewAI. You want your agents to access DB tables when the LLM tells them to, dynamically, thereby not having to dump pre-fetched DB data into user prompts. You will use the nl2sql tool of CrewAI as follows to create an internal tool.

The tool can be created as follows.

nl2sql_tool = NL2SQLTool(
    db_uri=database_uri
)

The database_uri is the connection string you have generated from the PostgreSQL DB itself to give read-only access to your agent_readonly user, as follows.

database_uri = (
    "postgresql+psycopg2://agent_readonly:secret123@localhost:5432/sales_db"
)

That’s how internal tools work.

B) External Tools

They are of two distinct types.

• Direct Tool Calling: These are the ones that you can directly access, often over API calls. For example, Google search API being called by your agent to directly access search results is a great example of a directly accessible external tool.
• Indirect Tool Calling: This is what we call MCP (Model-Context Protocol), popularized in late 2024 by Anthropic. In this case, a tool is hosted on somebody else’s server. Your agent’s job is to dynamically discover the tool registry at that URL via API requests and pass it on to the LLM. The LLMs then decide which one to use to get the job done. For example, an MCP server might host 100 tools, out of which you may need only one tool: europe_weather_paid_v2.

Part-03: The RAG Layer

— Equipping the Agentic Harness with Semantic Searching Capability Across Your Organization’s Static Assets —

Organizations have assets of two types:

• Structured data — SQL databases, NoSQL databases, etc.
• Unstructured data — images, GIFs, videos, documents, etc. stored in the object store.

RAG, which stands for Retrieval Augmented Generation, is the process of searching information inside the organization’s unstructured static assets.

The way we bring this about is by following the entire RAG pipeline, which contains the following steps: asset loading, parsing, chunking, embedding, storage, retrieval, re-ranking and finally ingestion into the LLM.

To understand RAG deeply, refer to the following articles written by me (0s & 1s — All About Software):

• Generative AI (Part-02): Retrieval Augment Generation (RAG)
• Generative AI (Part-05): RAG or Direct LLM Prompting?
• Generative AI (Part-07): Chunked Vs. Chunk-less RAG

Part-04: The Observability Layer

— Trace Workflows, Evaluate Performance & Improve Iteratively —

When the framework abstracts away everything in the tools, RAG and orchestration layer from us, we need a mechanism to monitor our application. A well-structured observability layer massively helps us with monitoring, tracing, logging, evaluating and improving upon our AI agents.

In particular, we monitor and log the following actions of agents.

— Inter-agent communication
— Agent-to-LLM communication
— Agent internal tool calls
— MCP tool calls
— Workflow latency
— Workflow failure rate
— Guardrail violation rate
— Total token usage per stage and per workflow
— Total cost to execute a single workflow
— CPU time used by the agents

… and much, much more.

In order to learn even better the observability hassle of the agentic apps, refer to this article by me (0s & 1s):

• AI Agents (05): The “Double Trouble” of Monitoring AI Agents

Part-05: The Cloud & Model Ecosystem Layer

— Beware the Pitfalls of Vendor Lock-in & Cloud Egress Fees! —

Each agentic framework comes packaged with what I call “invisible baggage”. The baggage is basically the “ecosystem / cloud / model trap”. In other words, big tech companies give away the entire framework and SDK for free — literally! — because they want to hook you into their cloud ecosystem.

A) Big Tech’s “Free” Frameworks are Basically “Hooks”

Take this for example.

• Google ADK, not surprisingly, tightly integrates with Gemini and GCP.
• Microsoft Agentic Framework tighthly integrates with .NET and Azure.
• OpenAI’s SDK tighthly integrates with the Azure-ChatGPT ecosystem.
• Bedrock Agents by AWS tighthly integrates with — guess what? — Amazon cloud.

…. and the examples go on and on. Hopefully, you get the idea.

B) Tech Stack, Cloud & Ecosystem Familiarity

Basically, keep this advice in mind which was given to me by my senior.

“When multiple frameworks meet your functional and non-functional requirements, it all then comes down to your internal tech stack and team’s familiarity and comfort with the ecosystem.” — My Senior’s Advice

That is very well put. Pretty much all the frameworks out there offer all the functionalities I haved shared in this article. Hence, the final decision often comes down to what cloud / model / ecosystem you are willing to get trapped in — lol.

C) Open-source Alternatives to Big Tech “Hooks”

Here are some open-source as well as paid alternatives to the big tech.

• LangGraph — A great agentic framework, integrates very well with LangSmith for observability.
• CrewAI — Another great open-source and free framework with a paid Crew cloud tier for agent observability. Use LangSmith with OTel for free-of-cost observability and to not get stuck in their cloud.
• Mastra — Yet another open-source framework with its own paid cloud observability layer.

The choice is your.

Summary — The ROOTS Acronym

Remember the “ROOTS” acronym, which stands for, if you haven’t already guessed:

• RAG
• Observability
• Orchestration
• Tools
• Skills: Read my article on agentic skills here; I skipped the discussion on skills in this article to prevent redundancy, but skills too are a cruicial part of any agentic harness / framework, and a discussion on them often surfaces in AI interviews and workloads.

Overall, in this long article, my goal was to bring to your awareness the first principles based on which you and your teams should be deciding what framework’s harness and cloud ecosystems are good for your agentic development needs.

Remember, there is no one, single right or wrong answer. Generally speaking, given that all the frameworks are offering all the features more or less, my advice is that you go with the framework that fits the best in your software and cloud ecosystem and whose language your team is the most familiar with — like Mastra for Next.js & Vercel ecosystem.

Until Next Time,
0s & 1s

Thank you for being a part of the community

Before you go:

Press enter or click to view image in full size

👉 Be sure to clap and follow the writer ️️
👉 Follow us on Medium
👉 CodeToDeploy Tech Community is live on Discord — join now!
👉 Disclosure: This post includes affiliate and partnership links.