LLM Agents, The Way Nobody Told You — Part 13: The Lessons
This is the final part of the series “LLM Agents, The Way Nobody Told You.” ◀ Part 12: How Do You Test an Agent?
This is the last article. I want to use it differently from the rest.
Everything so far was technical — the loop, tools, memory, sub-agents, hooks, caching, testing. The what and the how. This one is the what-now. The principles that survived the project, the decisions I would reverse with hindsight, and a few opinions I kept off the page until now.
If you have stuck with me through twelve articles, this is where I tell you what the whole thing actually taught me.
The principles worth keeping
These are the ideas that earned their place. I use them at work now, not just in agent code. They generalise.
Design observability when you build, not after.
The single most important principle from the series. The audit log in Part 7 did not exist because I needed testing — it existed because hooks needed to record every gated decision. Then in Part 12, the entire testing harness fell out of it almost for free.
You cannot bolt observability onto a system that was not built for it. What you can observe later is fixed by the architecture you choose now. Build every system to emit structured events from day one, even when you do not yet know what they are for. Future you will thank present you.
Stable contracts, evolving implementations.
The core data types in the project did not change once across all the phases. The implementations behind them changed constantly — single-shot to tools, in-process to MCP, prose to structured results. The interface stayed. The engine swapped underneath.
This is the highest-leverage habit in modern engineering. Every interface you write is either a stable contract — small, deliberate, change-resistant — or a leaky implementation detail that sprawls and couples everything together. The first kind compounds. The second kind rots.
Trade latency for grounding.
Every tool call replaces a chance to hallucinate with a verified fact. The single-shot version was fast and lying. The tool version was a little slower and correct. Correct and slightly slower beats fast and wrong for any customer-facing system, every time.
I have watched people optimise agents the wrong way — cutting tool calls to make it feel snappier — and ship agents that lie to users faster. The extra time is the agent checking its facts instead of guessing. That is time well spent.
Agents are tools.
Wrap everything — sub-agents, external APIs, side effects, database queries — as tool calls. Same mechanism, same hook layer, same audit log, same test surface.
The question in agentic systems is never “should this be a tool?” It is “is there a reason this is not a tool?” Direct Python calls give you none of the observability, gating, or composability the tool surface gives you for free.
The output is not proof. The trajectory is proof.
The agent that fluently describes work it never did. The reply that sounds right while nothing actually happened underneath. This is the production bug that defines agent engineering. If you remember nothing else from this series, remember this one.
Capability, authorisation, policy — different layers.
This took me longest to internalise. Tools check shape — are these inputs valid. Hooks check permission and policy — are you allowed to do this right now. They are different layers because they need different inputs. Tools only see what the model generated. Hooks also see the trusted server-side truth. Put the right check in the right layer and the whole codebase gets cleaner in a way that is hard to describe until you feel it.
Decisions I would reverse
Being honest about what I would change.
I should have built the audit log in Part 4, not Part 7.
Until Part 7, my logging was print statements. When I finally built the audit hook, I was retrofitting observability into an agent that had none. It worked, but I had spent several phases debugging blind.
The audit log is so cheap to add and so valuable to have that it should have been one of the first things I built. If you are starting your own agent: add structured logging in week one. Not later.
I underestimated how much of this is API contract management.
I thought agents would be about prompts and reasoning. Most of the actual difficulty turned out to be protocol mechanics — the tool_use and tool_result binding, the alternating-roles rule, the BLOCKED prefix convention, the byte-exact cache rule, the streaming timing rules.
None of that is prompting. It is the boring engineering layer underneath the magic, and it is almost the entire job. If I started over, I would read the tool-use, streaming, and prompt-caching sections of the API reference closely before writing any code — that is where the contracts that actually bite you live.
I should have built a tiny testing harness early.
The harness in Part 12 caught real bugs. But a three-case version right after the agent loop existed would have caught earlier bugs earlier. Trajectory testing does not need MCP, hooks, or sub-agents to be useful. It needs an audit log, a few expected behaviours, and a small scorer. Ship it early even if it is tiny.
I should have used streaming from the start.
I added streaming as an optimisation near the end. It turned out to be more than that — the agent felt alive, the experience improved, and debugging got easier because I could watch output as it streamed. It is not an optimisation. It is a default.
Get Simran Kaur Kahlon’s stories in your inbox
Join Medium for free to get updates from this writer.
I would be more aggressive about deleting flexibility.
Early on I wrote a lot of code that was flexible in case future needs changed. Most of that flexibility was never used and some of it made the code harder to follow. The phases where I wrote the most opinionated, least-flexible code aged the best. You are not gonna need it applies harder to agent code than most, because the problem space moves so fast that future-proofing guesses are usually wrong.
Opinions you might not have heard
These are takes I have not seen written elsewhere. I could be wrong about any of them. This is just how I think after building the thing.
Frameworks are mostly a distraction when you are learning.
I built the whole agent without any framework — raw SDK, my own Python, MCP. The result was easier to understand, debug, evolve, and teach. If you are learning, build from the SDK first. You will understand what frameworks abstract over, which means you can actually judge whether you need one. Most people would be served by a few hundred lines of well-organised Python and an MCP server.
The loop is the thing. Everything else is plumbing.
Once you understand the four-step loop, you understand the heart of every agent. Frameworks, orchestration, multi-agent systems — all variations on the same loop with different things wrapped around it. If you can write the loop on a whiteboard, you can read any framework’s source and understand it in five minutes. If you cannot, every framework feels like magic. Get the loop into your bones.
Sub-agents are the right idea. Most “multi-agent systems” are not.
The orchestrator and specialist pattern is genuinely useful — focused prompts, scoped tools, cost-aware routing. What people often call multi-agent systems — agents arguing over a shared scratchpad, electing leaders, voting on decisions — looks impressive in demos and rarely ships. The real constraint is almost always “this agent needs to do specific work reliably,” and the answer is almost always one orchestrator, a few specialists, structured handoffs. Build boring. Boring ships.
Most agents have more tools than they need.
I started with twelve tools and ended with eleven, three of which I would happily delete. The model makes better decisions with a short, unambiguous tool list than with fifteen overlapping options. More tools means more decision burden means more confused routing. If you cannot articulate why a tool exists, delete it.
The verdict field is one of the most useful patterns in agent engineering and almost nobody talks about it.
Structured returns with a categorical verdict field that your code branches on, separate from the natural-language explanation. This one pattern eliminates endless “LLM parses LLM output” round-trips. It cuts API calls, cuts latency, makes the system deterministic where it should be, and makes real testing possible because you can assert on a verdict instead of parsing English. I think a lot of the next few years of agent improvements will look like replacing English with structured discriminators in more and more places.
If you are just starting
If you are about to build your first agent, here is the order I would recommend:
- Start with one API call and a structured schema. Reasoning before answer.
- Build the four-step loop by hand. No frameworks. Print the messages list. Understand every block.
- Add memory by moving the messages list onto an object. It is that simple.
- Add an audit log. Now. Every tool call appends a record.
- Add three test cases that read the audit log and check the right tools fired.
- Build sub-agents. Wrap them as tools. Use the verdict pattern.
- Add write tools with the case-file pattern — store-assigned IDs, append-only.
- Move tools to an MCP server, while they are still small.
- Add hooks for authorisation.
- Add caching and streaming. Not later.
- Grow your test suite as you go. Do not backfill it at the end.
Each step adds one capability and answers one question. By the end you have a real agent and, more importantly, a real understanding of every layer in it.
A note on learning
I will close with something less technical.
I built this project because I learn by building. I wrote this series because I learn by explaining. Writing the articles taught me as much as building the agent did, maybe more. Putting concepts into words is a different operation than running them through code. Both are necessary.
If you have been reading this passively, I want to push back. You will not understand agents from reading about them — even from reading me. You will understand them from writing a tool definition, watching the model call it wrong, debugging why, and fixing it. Then doing that fifty more times.
The articles are a map. The territory is your terminal.
Pick a domain — support, scheduling, research, anything with multiple intents. Mock the backend. Write the loop yourself. Make every mistake I made plus the ones I missed. Then come back and re-read whichever article covers the thing you got stuck on. The second read will land in a way the first one could not.
Thank you for reading the whole series. Long-form technical writing only matters when someone reads it. Knowing this helped even one engineer trying to make sense of agents is why I will keep writing.
Now go build something.
This is the final part of “LLM Agents, The Way Nobody Told You.” ◀ Part 12: How Do You Test an Agent?
