I recently worked on a feature where I had to transform 100 messy compliance pdfs into structured JSON rules.

The brute force approach was obvious: give the agent the source text, explain the task, provide examples, and ask it to generate the rules. Since it was the lowest-hanging fruit, I tried it first.

At a glance, the output looked fine. The output JSON was valid and matched what I expected.

But as I was manually sampling the results to check for accuracy, the cracks appeared. Some rules were too broad, others were missed. Some rules failed to preserve the nuances of the original text. I tried using another agent to catch and fix the errors but with such a huge corpus, it was impossible to confidently verify the output.

That was the frustrating part. The errors were not obvious. This was way too fragile of an implementation to scale.

Though I cannot share the exact implementation details, what I can share are the architectural lessons I learnt and how I eventually implemented it. Hopefully, these insights will be useful if you’re building AI systems that need to scale, stay reliable, and deal with messy data. And if you have better ways of doing things, do reach out to chat!

Okay let’s get to it.

The problem

The 100 pdfs I worked with had already been parsed and chunked before they reached me. But the raw content was still messy. There were bullet points, tables, OCR artefacts, translated sections, semi structured headings, footers, headers, inconsistent formatting and document specific quirks.

I chose to use an agent because deciding what mattered required semantic judgement. The documents did not follow one consistent pattern, so relevance could not be determined through simple rules alone.

You had to understand the surrounding context. None of this was difficult when done on a small chunk of data. The challenge was performing this reliably at scale.

These rules were then processed by another downstream system to be evaluated deterministically.

What eventually worked

After a few experiments, I realised the biggest improvement did not come from a better prompt, a new tool, an MCP server, or a more sophisticated agent harness.

It came from changing the shape of the problem.

Instead of trying to make the agent smarter, I made the agent’s job smaller.

The first change was to prepare the source data upfront. Instead of asking the agent to query a database, retrieve records, decide whether it had the right inputs, and then perform the extraction, I gave it a more controlled starting point.

In my case, that meant temporarily storing the relevant raw data locally.

This may not always be practical. But the underlying principle is to reduce the amount of retrieval uncertainty the agent has to handle. If the agent’s job is to reason over content, do not also make it responsible for figuring out whether it has found the right content.

Another option would be to prepare the query upfront.

I also used a script to strip away unnecessary metadata and fields before passing the raw content to the agent. Less irrelevant context meant fewer distractions, fewer chances for the agent to latch onto the wrong details and a cleaner reasoning task overall.

But the most important change was the unit of work.
Instead of processing everything at once, I did things iteratively and processed one document at a time.

That made each job smaller, easier to inspect, easier to retry, and easier to audit. I spun up five subagents to process documents in parallel, with each agent logging its progress to a file.

If one document failed, I could retry only that document. If one output had formatting issues, I could fix that specific case without rerunning the whole batch. If the pipeline stopped halfway, the cached progress meant it could resume from the last successful checkpoint.

This was also where the separation of responsibilities became clearer.

The agent handled the semantic work: understanding the content, identifying the relevant parts and writing the JSON output.

The surrounding code handled the mechanical parts: parallelising jobs, enforcing the schema, generating IDs, writing files, caching progress, validating references, and checking whether the output could be traced back to the original source.

I also had an orchestrator watch over the progress of the script.

Making the output auditable

A useful design decision was adding reference IDs to every generated rule. This meant that each output item pointed back to a specific source.

This made the output easier to audit. Instead of asking, “Does this generated rule look right?”, I could ask more precise questions such as: does the referenced source chunk exist? Is the quoted source text actually present in that chunk?

I could also get another agent to selectively run audits on larger and more complex documents to ensure that important nuances were preserved.

On top of that, I did a lightweight version of evals. I ran a small batch of raw documents through the workflow and manually reviewed the results for coverage and accuracy. A full golden dataset was not practical for the scope of this task, but I still needed a way to prove to myself that the workflow was working.

My goal was not to build a perfect benchmark but to make the system auditable enough that I could inspect the outputs, catch failures, and iterate toward a higher accuracy bar.

If you’ve got ideas on how I could have done this better, let me know!

My biggest takeaway

The pattern that worked was to stop treating the LLM as the whole system.

The system became more reliable not because the agent became perfect, but because the workflow made its outputs easier to trace, validate, and recover from.

Coincidentally, I was building this shortly before attending the inaugural AI Engineer Singapore conference, held from 15–17 May 2026.

On the last day, JJ Geewax, Director of Applied AI at Google DeepMind, shared a framing that captured what I had been learning the hard way: we need to stop using LLMs like giant problem solvers.

That resonated with me because it is such an easy trap to fall into. It is easy to just give the model the data, schema, business rules, edge cases, and the responsibility to verify itself. Then get frustrated when the result is inconsistent.

But for reliable production systems, the better pattern is usually a hybrid. Let the agent handle the parts that require semantic judgement, and let code handle the parts that require structure, validation, and control.

I’ll be sharing more reflections from AI Engineer Singapore and the workshops I attended. The YouTube snippet of JJ’s speech here:

- YouTube

That’s all from me. I hope this helped, and see you in the next article :)

Stop using LLMs like giant problem solvers was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

We talk a lot about better models, larger context windows, and more capable agents. But most real-world failures don’t come from model capability — they come from how context is constructed, passed, and maintained.

This is a hard problem. The space is moving fast and techniques are still evolving. Much of it remains an experimental science and depends on the context (pun intended), constraints and environment you’re operating in.

In my work building multi-agent systems, a recurring pattern has emerged: performance is far less about how much context you give a model, and far more about how precisely you shape it.

This piece is an attempt to distill my learnings into something you can use.

It focuses on principles for managing context as a constrained resource — deciding what to include, what to exclude, and how to structure information so that agents remain coherent, efficient, and reliable over time.

Because at the end of the day, the strongest agents are not the ones that see the most. They are the ones that see the right things, in the right form, at the right time.

Terminology

Context engineering

Context engineering is the art of providing the right information, tools and format to an LLM for it to complete a task. Good context engineering means finding the smallest possible set of high signal tokens that give the LLM the highest probability of producing a good outcome.

In practice, good context engineering usually comes down to four moves. You offload information to external systems (context offloading) so the model does not need to carry everything in-band. You retrieve information dynamically instead of front-loading all of it (context retrieval). You isolate context so one subtask does not contaminate another (context isolation). And you reduce history when needed, but only in ways that preserve what the agent will still need later (context reduction).

A common failure mode on the other side is context pollution: the presence of too much unnecessary, conflicting or redundant information that it distracts the LLM.

Context rot

Context rot is a situation where an LLM’s performance degrades as the context window fills up, even if it is within the established limit. The LLM still has room to read more, but its reasoning starts to blur.

You would have noticed that the effective context window, where the model performs at high quality, is often much smaller than what the model technically is capable of.

There are two parts to this. First, a model does not maintain perfect recall across it’s entire context window. Information at the start and the end is more reliably recalled than things in the middle.

Second, larger context windows do not solve problems for enterprise systems. Enterprise data is effectively unbounded and frequently updated that even if the model could ingest everything, that would not mean it could maintain a coherent understanding over it.

Just like humans have a limited working memory capacity, every new token introduced to the LLM depletes this attention budget it has by some amount. The attention scarcity stems from architectural constraints in the transformer, where every token attends to every other token. This leads to a n² interaction pattern for n tokens. As the context grows, the model is forced to spread its attention thinner across more relationships.

Context compaction

Context compaction is the general answer to context rot.

When the model is nearing the limit of it’s context window, it summarises it’s contents and reinitiates a new context window with the previous summary. This is especially useful for long running tasks to allow the model to continue to work without too much performance degradation.

Recent work on context folding offers a different approach — agents actively manage their working context. An agent can branch off to handle a subtask and then fold it upon completion, collapsing the intermediate steps while retaining a concise summary of the outcome.

The difficulty, however, is not in summarising, but in deciding what survives. Some things should remain stable and nearly immutable, such as the objective of the task and hard constraints. Others can be safely discarded. The challenge is that the importance of information is often only revealed later.

Good compaction therefore needs to preserve facts that continue to constrain future actions: which approaches already failed, which files were created, which assumptions were invalidated, which handles can be revisited, and which uncertainties remain unresolved. Otherwise you get a neat, concise summary that reads well to a human and is useless to an agent.

Agent harness

A model is not an agent. The harness is what turns a model into one.

By harness, I mean everything around the model that decides how context is assembled and maintained: prompt serialization, tool routing, retry policies, the rules governing what is preserved between steps, and so on.

Once you look at real agent systems this way, a lot of supposed “model failures” now look different. I’ve encountered many of such at work. These are actually harness failures: the agent forgot because nothing persisted the right state; it repeated work because the harness surfaced no durable artefact of prior failure; it chose the wrong tool because the harness overloaded the action space; and so on.

A good harness is, in some sense, a deterministic shell wrapped around a stochastic core. It makes the context legible, stable, and recoverable enough that the model can spend its limited reasoning budget on the task rather than on reconstructing its own state from a messy trace.

Communication between agents

As tasks get more complex, teams have defaulted towards multi-agent systems.

The mistake is to assume that more agents means more shared context. In practice, dumping a giant shared transcript into every sub-agent often creates exactly the opposite of specialisation. Now every agent is reading everything, inheriting everyone else’s mistakes, and paying the same context bill over and over again.

If only some context is shared, a new problem appears. What is considered authoritative when agents disagree? What remains local, and how are conflicts reconciled?

The way out is to treat communication not as shared memory, but as state transfer through well-defined interfaces.

For discrete tasks with clear inputs and outputs, agents should usually communicate through artefacts rather than raw traces. A web-search agent, for instance, does not need to pass along its entire browsing history. It only needs to surface the material that downstream agents can actually use.

This means that intermediate reasoning, failed attempts, and exploration traces stay private unless explicitly needed. What gets passed forward are distilled outputs: extracted facts, validated findings, or decisions that constrain the next step.

For more tightly coupled tasks, like a debugging agent where downstream reasoning genuinely depends on prior attempts, a limited form of trace sharing can be introduced. But this should be deliberate and scoped, not the default.

KV cache penalty

When AI models generate text, they often repeat many of the same calculations. KV caching is an inference time optimisation technique that speeds up this process by remembering important information from previous steps instead of recomputing everything again.

However, in multi-agent systems, if every agent shares the same context, you confuse the model with a ton of irrelevant details and pay a massive KV-cache penalty. Multiple agents working on the same task need to communicate with each other, but this should not be via sharing memory.

This is why agents should communicate through minimal, structured outputs in a controlled manner.

Keep the agent’s toolset small and relevant

Tool choice is a context problem disguised as a capability problem.

As an agent accumulates more tools, the action space gets harder to navigate. There is now a higher probability of the model going down the wrong action and taking an inefficient route.

This has consequences. Tool schemas need to be far more distinct than most people realise. Tools have to be well understood and have minimal overlap in functionality. It should be very clear on what their intended use is and have clear input parameters that are unambiguous.

One common failure mode that I noticed even in my team is that we tend to have very bloated sets of tools that are added over time. This leads to unclear decision making on which tools to use.

Agentic memory

This is a a technique where the agent regularly writes notes persisted to memory outside of the context window. These notes get pulled back into the context window at later times.

The hardest part is deciding what deserves promotion into memory. My rule of thumb is that durable memory should contain things that continue to constrain future reasoning: persistent preferences. Everything else should have a very high bar. Storing too much is just another route back to context pollution, only now you have made it persistent.

But memory without revision is a trap. Once agents persist notes across steps or sessions, they also need mechanisms for conflict resolution, deletion, and demotion. Otherwise long-term memory becomes a landfill of outdated beliefs.

To sum up

Context engineering is still evolving, and there is no single correct way to do it. Much of it remains empirical, shaped by the systems we build and the constraints we operate under.

Left unchecked, context grows, drifts, and eventually collapses under its own weight.

If well-managed, context becomes the difference between an agent that merely responds and one that can reason, adapt, and stay coherent across long and complex tasks.

Thank you for staying with me till the end, and I hope this was useful for you. Subscribe to my content to follow along my journey.

Deep dive into context engineering for AI agents was originally published in Code Like A Girl on Medium, where people are continuing the conversation by highlighting and responding to this story.

LangChain shipped its first stable v1.0 release in late October 2025. After spending the past two months working with their new APIs, I genuinely feel this is the most coherent and thoughtfully designed version of LangChain to date.

I wasn’t always a LangChain fan. The early versions were fragile, poorly documented, abstractions shifted frequently, and it felt too premature to use in prod. But v1.0 felt more intentional, and had a more consistent mental model for how data should flow through agents and tools.

This isn’t a sponsored post btw — I’d love to hear your thoughts, so leave them in the comments or DM me here!

This article isn’t here to regurgitate the docs. I’m assuming you’ve already dabbled with LangChain (or are a heavy user). Rather than dumping a laundry list of points, I’m going to cherry-pick just four key points.

A quick recap: LangChain, LangGraph & LangSmith

At a high level, LangChain is a framework for building LLM apps and agents, allowing devs to ship AI features fast with common abstractions.

LangGraph is the graph based execution engine for durable, stateful agent workflows in a controllable way. Finally, LangSmith is an observability platform for tracing and monitoring.

Put simply: LangChain helps you build agents fast, LangGraph runs them reliably, and LangSmith lets you monitor and improve them in production.

My stack

For context, most of my recent work focuses on building multi-agent features for a customer-facing AI platform at work. My backend stack is FastAPI, with Pydantic powering schema validation and data contracts.

Lesson 1: Dropping support for Pydantic models

A major shift in the migration to v1.0 was the introduction of the new create_agent method. It streamlines how agents are defined and invoked, but it also drops support for Pydantic models and dataclasses in agent state. Everything must now be expressed as TypedDicts extending AgentState.

If you’re using FastAPI, Pydantic is often the recommended and default schema validator. I valued schema consistency across the codebase and felt that mixing TypedDicts and Pydantic models would inevitably create confusion — especially for new engineers who might not know which schema format to follow.

To solve this, I introduced a small helper that converts a Pydantic model into a TypedDict that extends AgentState right before it’s passed to create_agent . It is critical to note that LangChain attaches custom metadata to type annotations which you must preserve. Python utilities like get_type_hints() strip these annotations, meaning a naïve conversion won’t work.

Lesson 2: Deep agents, default middleware & the lack of control

Alongside the new create_agent API in LangChain 1.0 came something that caught my attention: the deepagentslibrary. Inspired by tools like Claude Code and Manus, deep agents can plan, break tasks into steps, and even spawn subagents.

When I first saw this, I wanted to use it everywhere. Why wouldn’t you want “smarter” agents, right? But after trying it across several workflows, I realised that this extra autonomy was sometimes unnecessary — and in certain cases, counterproductive — for my use cases.

Each deep agent comes with some built-in middleware — things like ToDoListMiddleware, FilesystemMiddleware, SummarizationMiddleware, etc. These shape how the agent thinks, plans, and manages context. The issue is that, at the time of writing, you can’t control when these middleware run, nor can you disable the ones you don’t need.

After digging into the deepagents source code here, you can see that the middleware parameter is additional middleware to apply after standard middleware. Any middleware that was passed in middleware=[...] gets appended after the defaults.

All this extra orchestration also introduced noticeable latency, without providing meaningful benefits for my specific workloads.

I’m not saying deep agents are bad, they’re powerful in the right scenarios. However, this is a good reminder of a classic engineering principle: don’t chase the “shiny” thing. Use the tech that solves your actual problem, even if it’s the “less glamorous” option.

My favourite feature: Structured output

Having deployed agents in production, especially ones that integrate with deterministic enterprise systems, getting agents to consistently produce output of a specific schema was crucial.

LangChain 1.0 makes this pretty easy. You can define a schema (e.g., a Pydantic model) and pass it to create_agent via the response_format parameter. The agent then produces output that conforms to that schema within a single agent loop with no additional steps.

This has been incredibly useful whenever I need the agent to strictly adhere to a JSON structure with certain fields guaranteed. So far, structured output has been very reliable too.

What I want to explore more of: Middleware

One of the trickiest parts of building reliable agents is context engineering — making sure the agent always has the right information at the right time. Middleware was introduced to give developers precise control over each step of the agent loop, and I think it is worth diving deeper into.

Middleware can mean different things depending on context (pun intended). In LangGraph, this can mean controlling the exact sequence of node execution. In long-running conversations, it might involve summarising accumulated context before the next LLM call. In human-in-the-loop scenarios, middleware can pause execution and wait for a user to approve or reject a tool call.

More recently, in the latest v1.1 minor release, LangChain also added a model retry middleware with configurable exponential backoff, allowing graceful recovery for transient endpoint errors.

I personally think middleware is a game changer as agentic workflows get more complex, long-running, and stateful, especially when you need fine-grained control or robust error handling.

This list of middleware is growing and it really helps that it remains provider agnostic. If you’ve experimented with middleware in your own work, I’d love to hear what you found most useful!

To end off

That’s it for now — four key reflections from what I’ve learnt so far about LangChain. And if anyone from the LangChain team happens to be reading this, I’m always happy to share user feedback anytime or simply chat :)

Have fun building!

Lessons Learnt From Upgrading to LangChain 1.0 in Production was originally published in Women in Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

My job doesn’t look quite the same as before. As a software engineer in the AI space, my work has been a hybrid of software engineering, AI engineering, product intuition, and doses of user empathy.

With so much going on, I wanted to take a step back and reflect on the bigger picture and the kind of skills and mental models engineers need to stay ahead. A recent read of O’Reilly’s AI Engineering gave me the nudge to also want to deep dive into how to think about evals — a core component in any AI system.

One thing stood out: AI engineering is often more software than AI.

Outside of research labs like OpenAI or Anthropic, most of us aren’t training models from scratch. The real work is about solving business problems with the tools we already have — giving models enough relevant context, using APIs, building RAG pipelines, tool-calling — all on top of the usual SWE concerns like deployment, monitoring and scaling.

In other words, AI engineering isn’t replacing software engineering — it’s layering new complexity on top of it.

This piece is me teasing out some of those themes. If any of them resonates, I’d love to hear your thoughts — feel free to reach out here!

The three layers of an AI application stack

Think of an AI app as being built on three layers: 1) Application development 2) Model development 3) Infrastructure.

Most teams start from the top. With powerful models readily available off the shelf, it often makes sense to begin by focusing on building the product and only later dip into model development or infrastructure as needed.

As O’Reilly puts it, “AI engineering is just software engineering with AI models thrown into the stack.”

Why evals matter and why they’re tough

In software, one of the biggest headaches for fast-moving teams is regressions. You ship a new feature, and in the process unknowingly break something else. Weeks later, a bug surfaces in a dusty corner of the codebase, and tracing it back becomes a nightmare.

Having a comprehensive test suite helps catch these regressions.

AI development faces a similar problem. Every change — whether it’s prompt tweaks, RAG pipeline updates, fine-tuning, or context engineering — can improve performance in one area while quietly degrading another.

In many ways, evaluations are to AI what tests are to software: they catch regressions early and give engineers the confidence to move fast without breaking things.

But evaluating AI isn’t straightforward. Firstly, the more intelligent models become, the harder evaluation gets. It’s easy to tell if a book summary is bad if it’s gibberish, but much harder if the summary is actually coherent. o know whether it’s actually capturing the key points, not just sounding fluent or factually correct, you might have to read the book yourself.

Secondly, tasks are often open-ended. There’s rarely a single “right” answer and impossible to curate a comprehensive list of correct outputs.

Thirdly, foundation models are treated as black boxes, where details of model architecture, training data and training process are often scrutinised or even made public. These details reveal alot about a model’s strengths and weaknesses and without it, people only evaluate models based by observing it’s outputs.

How to think about evals

I like to group evals into two broad realms: quantitative and qualitative.

Quantitative evals have clear, unambiguous answers. Did the math problem get solved correctly? Did the code execute without errors? These can often be tested automatically, which makes them scalable.

Qualitative evals, on the other hand, live in the grey areas. They’re about interpretation and judgment — like grading an essay, assessing the tone of a chatbot, or deciding whether a summary “sounds right.”

Most evals are a mix of both. For example, evaluating a generated website means not only testing whether it performs its intended functions (quantitative: can a user sign up, log in, etc.), but also judging whether the user experience feels intuitive (qualitative).

Functional correctness

At the heart of quantitative evals is functional correctness: does the model’s output actually do what it’s supposed to do?

If you ask a model to generate a website, the core question is whether the site meets its requirements. Can a user complete key actions? Does it work reliably? This looks a lot like traditional software testing, where you run a product against a suite of test cases to verify behaviour. Often, this can be automated.

Similarity against reference data

Not all tasks have such clear, testable outputs. Translation is a good example: there’s no single “correct” English translation for a French sentence, but you can compare outputs against reference data.

The downside: This relies heavily on the availability of reference datasets, which are expensive and time-consuming to create. Human-generated data is considered the gold standard, but increasingly, reference data is being bootstrapped by other AIs.

There are a few ways to measure similarity:

• Human judgement
• Exact match: whether the generated response matches one of the reference responses exactly. These produces boolean results.
• Lexical similarity: measuring how similar the outputs look (e.g., overlap in words or phrases).
• Semantic similarity: measuring whether the outputs mean the same thing, even if the wording is different. This usually involves turning data into embeddings (numerical vectors) and comparing them. Embeddings aren’t just for text — platforms like Pinterest use them for images, queries, and even user profiles.

Lexical similarity only checks surface-level resemblance, while semantic similarity digs deeper into meaning.

AI as a judge

Some tasks are nearly impossible to evaluate cleanly with rules or reference data. Assessing the tone of a chatbot, judging the coherence of a summary, or critiquing the persuasiveness of ad copy all fall into this category. Humans can do it, but human evals don’t scale.

Here’s how to structure the process:

1. Define a structured and measurable evaluation criteria. Be explicit about what you care about — clarity, helpfulness, factual accuracy, tone, etc. Criteria can use a scale (1–5 rating) or binary checks (pass/fail).
2. The original input, the generated output, and any supporting context are given to the AI judge. A score, label or even an explanation for evaluation is then returned by the judge.
3. Aggregate over many outputs. By running this process across large datasets, you can uncover patterns — for example, noticing that helpfulness dropped 10% after a model update.

Because this can be automated, it enables continuous evaluation, borrowing from CI/CD practices in software engineering. Evals can be run before and after pipeline changes (from prompt tweaks to model upgrades), or used for ongoing monitoring to catch drift and regressions.

Of course, AI judges aren’t perfect. Just as you wouldn’t fully trust a single person’s opinion, you shouldn’t fully trust a model’s either. But with careful design, multiple judge models, or running them over many outputs, they can provide scalable approximations of human judgment.

Eval driven development

O’Reilly talked about the concept of eval-driven development, inspired by test-driven development in software engineering, something I felt is worth sharing.

The idea is simple: Define your evals before you build.
In AI engineering, this means deciding what “success” looks like and how it’ll be measured.

Impact still matters most — not hype. The right evals ensure that AI apps demonstrate value in ways that are relevant to users and the business.

When defining evals, here are some key considerations:

Domain knowledge

Public benchmarks exist across many domains — code debugging, legal knowledge, tool use — but they’re often generic. The most meaningful evals usually come from sitting down with stakeholders and defining what truly matters for the business, then translating that into measurable outcomes.

Correctness isn’t enough if the solution is impractical. For example, a text-to-SQL model might generate a correct query, but if it takes 10 minutes to run or consumes huge resources, it’s not useful at scale. Runtime and memory usage are important metrics too.

Generation capability

For generative tasks — whether text, image, or audio — evals may include fluency, coherence, and task-specific metrics like relevance.

A summary might be factually accurate but miss the most important points — an eval should capture that. Increasingly, these qualities can themselves be scored by another AI.

Factual consistency

Outputs need to be checked against a source of truth. This can happen in two ways:

1. Local consistency
   This means verifying outputs against a provided context. This is especially useful for specific domains that are unique to themselves and have limited scope. For instance, extracted insights should be consistent with the data.
2. Global consistency
   This means verifying outputs against open knowledge sources such as by fact checking via a web search or a market research and so on.
3. Self verification
   This happens when a model generates multiple outputs, and measures how consistent these responses are with each other.

Safety

Beyond the usual concept of safety such as to not include profanity and explicit content, there are actually many ways in which safety can be defined. For instance, chatbots should not reveal sensitive customer data and should be able to guard against prompt injection attacks.

To sum up

As AI capabilities grow, robust evals will only become more important. They’re the guardrails that let engineers move quickly without sacrificing reliability.

I’ve seen how challenging reliability can be and how costly regressions are. They damage a company’s reputation, frustrate users, and create painful dev experiences, with engineers stuck chasing the same bugs over and over.

As the boundaries between engineering roles blur, especially in smaller teams, we’re facing a fundamental shift in how we think about software quality. The need to maintain and measure reliability now extends beyond rule-based systems to those that are inherently probabilistic and stochastic.

AI Engineering and Evals as New Layers of Software Work was originally published in Code Like A Girl on Medium, where people are continuing the conversation by highlighting and responding to this story.

If you’re an engineer, have worked with machine learning or large-scale data pipelines, chances are you’ve used some sort of queueing system.

Queues let services talk to each other asynchronously: you send off work, don’t wait around, and let another system pick it up when ready. This is essential when your tasks aren’t instant — think long-running model training jobs, batch ETL pipelines, or even processing requests for LLMs that take minutes per query.

So why am I writing this? I recently migrated a production queueing setup to RabbitMQ, ran into a bunch of bugs, and found that documentation was thin on the trickier parts. After a fair bit of trial and error, I thought it’d be worth sharing what I learned.

Hope you will find this useful!

A quick primer: queues vs request-response model

Microservices typically communicate in two styles — the classic request–response model, or the more flexible queue-based model.

Imagine ordering pizza. In a request–response model, you tell the waiter your order and then wait. He disappears, and thirty minutes later your pizza shows up — but you’ve been left in the dark the whole time.

In a queue-based model, the waiter repeats your order, gives you a number, and drops it into the kitchen’s queue. Now you know it’s being handled, and you’re free to do something else till the chef gets to it.

That’s the difference: request–response keeps you blocked until the work is done, while queues confirm right away and let the work happen in the background.

What is Rabbit MQ?

RabbitMQ is a popular open-source message broker that ensures messages are reliably delivered from producers (senders) to consumers (receivers). First released in 2007 and written in Erlang, it implements AMQP (Advanced Message Queuing Protocol), an open standard for structuring, routing, and acknowledging messages.

Think of it like a post office for distributed systems: applications drop off messages, RabbitMQ sorts them into queues, and consumers pick them up when ready.

A common pairing in the Python world is Celery + RabbitMQ: RabbitMQ brokers the tasks, while Celery workers execute them in the background.

In containerised setups, RabbitMQ typically runs in its own container, while Celery workers run in separate containers that you can scale independently.

How it works at a high level

Your app wants to run some work asynchronously. Since this task might take a while, you don’t want the app to sit idle waiting. Instead, it creates a message describing the task and sends it to RabbitMQ.

1. Exchange: This lives inside RabbitMQ. It doesn’t store messages but just decides where each message should go based on rules you set (routing keys and bindings).
   Producers publish messages to an exchange, which acts as a routing intermediary.
2. Queues: They’re like mailboxes. Once the exchange decides which queue(s) a message should go to, it sits there till it’s picked up.
3. Consumer: The service that reads and processes messages from a queue. In a Celery setup, the Celery worker is the consumer — it pulls tasks off the queue and does the actual work.

Once the message is routed into a queue, the RabbitMQ broker pushes it out to a consumer (if one is available) over a TCP connection.

Core components in Rabbit MQ

1. Routing and binding keys

Routing and binding keys work together to decide where a message ends up.

• A routing key is attached to a message by the producer.
• A binding key is the rule a queue declares when it connects (binds) to an exchange.
  A binding defines the link between an exchange and a queue.

When a message is sent, the exchange looks at the message’s routing key. If that routing key matches the binding key of a queue, the message is delivered to that queue.

A message can only have one routing key.
A queue can have one or multiple binding keys, meaning it can listen for several different routing keys or patterns.

2. Exchanges

An exchange in RabbitMQ is like a traffic controller. It receives messages, does not store messages, and it’s key job is to decide which queue(s) the message should go to, based on rules.

If the routing key of a message does not match any the binding keys of any queues, it will not get delivered.

There are several types of exchanges, each with its own routing style.

2a) Direct exchange

Think of a direct exchange like an exact address delivery. The exchange looks for queues with binding keys that exactly match the routing key.

• If only one queue matches, the message will only be sent there (1:1).
• If multiple queues have the same binding key, the message will be copied to all of them (1:many).

2b) Fanout exchange

A fanout exchange is like shouting through a loudspeaker.

Every message is copied to all queues bound to the exchange. The routing keys are ignored, and it is always a 1:many broadcast.

Fanout exchanges can be useful when the same message needs to be sent to one or more queues with consumers who may process the same message in different ways.

2c) Topic exchange

A topic exchange works like a subscription system with categories.

Every message has a routing key, for example "order.completed”. Queues can then subscribe to patterns such as "order.*”. This means that whenever a message is related to an order, it will be delivered to any queues that have subscribed to that category.

Depending on the patterns, a message might end up in just one queue or in several at the same time.

There are two important special cases for binding keys:

• * (star) matches exactly one word in the routing key.
• # (hash) matches zero or more words.

Let’s illustrate this to make the syntax alot more intuitive.

2d) Headers exchange

A headers exchange is like sorting mail by labels instead of addresses.

Instead of looking at the routing key (like "order.completed"), the exchange inspects the headers of a message: These are key–value pairs attached as metadata. For instance:

• x-match: all, priority: high, type: email → the queue will only get messages that have both priority=high and type=email.
• x-match: any, region: us, region: eu → the queue will get messages where at least one of the conditions is true (region=us or region=eu).

The x-match field is what determines whether all rules must match or any one rule is enough.

Because multiple queues can each declare their own header rules, a single message might end up in just one queue (1:1) or in several queues at once (1:many).

Headers exchanges are less common in practice, but they’re useful when routing depends on more complex business logic. For example, you might want to deliver a message only if customer_tier=premium, message_format=json, or region=apac .

2e) Dead letter exchange

A dead letter exchange is a safety net for undeliverable messages.

3. A push delivery model

This means that as soon as a message enters a queue, the broker will push it out to a consumer that is subscribed and ready. The consumer doesn’t request messages and instead just listens on the queue.

This push approach is great for low-latency delivery — messages get to consumers as soon as possible.

Useful features in Rabbit MQ

Rabbit MQ’s architecture lets you shape message flow to fit your workload. Here are some useful patterns.

Work queues — competing consumers pattern

You publish tasks into one queue, and many consumers (eg. celery workers) all listen to that queue. The broker delivers each message to exactly one consumer, so workers “compete” for work. This implicitly translates to simple load-balancing.

If you’re on celery, you’ll want to keep worker_prefetch_multiplier=1 . What this means is that a worker will only fetch one message at a time, avoiding slow workers from hoarding tasks.

Pub/sub pattern

Multiple queues bound to an exchange and each queue gets a copy of the message (fanout or topic exchanges). Since each queue gets its own message copy, so different consumers can process the same event in different ways.

Explicit acknowledgements

RabbitMQ uses explicit acknowledgements (ACKs) to guarantee reliable delivery. An ACK is a confirmation sent from the consumer back to the broker once a message has been successfully processed.

When a consumer sends an ACK, the broker removes that message from the queue. If the consumer NACKs or dies before ACKing, RabbitMQ can redeliver (requeue) the message or route it to a dead letter queue for inspection or retry.

There is, however, an important nuance when using Celery. Celery does send acknowledgements by default, but it sends them early — right after a worker receives the task, before it actually executes it. This behaviour (acks_late=False, which is the default) means that if a worker crashes midway through running the task, the broker has already been told the message was handled and won’t redeliver it.

Priority queues

RabbitMQ has a out of the box priority queueing feature which lets higher priority messages jump the line. Under the hood, the broker creates an internal sub-queue for each priority level defined on a queue.

For example, if you configure five priority levels, RabbitMQ maintains five internal sub-queues. Within each level, messages are still consumed in FIFO order, but when consumers are ready, RabbitMQ will always try to deliver messages from higher-priority sub-queues first.

Doing so implicitly would mean an increasing amount of overhead if there were many priority levels. Rabbit MQ’s docs note that though priorities between 1 and 255 are supported, values between 1 and 5 are highly recommended.

Message TTL & scheduled deliveries

Message TTL (per-message or per-queue) automatically expires stale messages; and delayed delivery is available via plugins (e.g., delayed-message exchange) when you need scheduled execution.

How to optimise your Rabbit MQ and Celery setup

When you deploy Celery with RabbitMQ, you’ll notice a few “mystery” queues and exchanges appearing in the RabbitMQ management dashboard. These aren’t mistakes — they’re part of Celery’s internals. After a few painful rounds of trial and error, here’s what I learned about how Celery really uses RabbitMQ under the hood — and how to tune it properly.

Kombu

Celery relies on Kombu, a Python messaging framework. Kombu abstracts away the low-level AMQP operations, giving Celery a high-level API to:

• Declare queues and exchanges
• Publish messages (tasks)
• Consume messages in workers

It also handles serialisation (JSON, Pickle, YAML, or custom formats) so tasks can be encoded and decoded across the wire.

Celery events and the celeryev Exchange

Celery includes an event system that tracks worker and task state. Internally, events are published to a special topic exchange called celeryev.

There are two such event types:

1. Worker events eg.worker.online, worker.heartbeat, worker.offline are always on and are lightweight liveliness signals.
2. Task events, eg.task-received, task-started, task-succeeded, task-failed which are disabled by default unless the -E flag is added.

You have fine grain control over both types of events. You can turn off worker events (by turning off gossip, more on that below) while turning on task events.

Gossip

Gossip is Celery’s mechanism for workers to “chat” about cluster state — who’s alive, who just joined, who dropped out, and occasionally elect a leader for coordination. It’s useful for debugging or ad-hoc cluster coordination.

By default, Gossip is enabled. When a worker starts:

• It creates an exclusive, auto-delete queue just for itself.
• That queue is bound to the celeryev topic exchange with the routing key pattern worker.#.

Because every worker subscribes to every worker.* event, the traffic grows quickly as the cluster scales.

With N workers, each one publishes its own heartbeat, and RabbitMQ fans that message out to the other N-1 gossip queues. In effect, you get an N × (N-1) fan-out pattern.

In my setup with 100 workers, that meant a single heartbeat was duplicated 99 times. During deployments — when workers were spinning up and shutting down, generating a burst of join, leave, and heartbeat events — the pattern spiraled out of control. The celeryev exchange was suddenly handling 7–8k messages per second, pushing RabbitMQ past its memory watermark and leaving the cluster in a degraded state.

When this memory limit is exceeded, RabbitMQ blocks publishers until usage drops. Once memory falls back under the threshold, RabbitMQ resumes normal operation.

However, this means that during the memory spike the broker becomes unusable — effectively causing downtime. You won’t want that in production!

The solution is to disable Gossip so workers don’t bind to worker.#. You can do this in the docker compose where the workers are spun up.

celery -A myapp worker --without-gossip

Mingle

Mingle is a worker startup step where the new worker contacts other workers to synchronise state — things like revoked tasks and logical clocks. This happens only once, during worker boot. If you don’t need this coordination, you can also disable it with --without-mingle

Occasional connection drops

In production, connections between Celery and RabbitMQ can occasionally drop — for example, due to a brief network blip. If you have monitoring in place, you may see these as transient errors.

The good news is that these drops are usually recoverable. Celery relies on Kombu, which includes automatic connection retry logic. When a connection fails, the worker will attempt to reconnect and resume consuming tasks.

As long as your queues are configured correctly, messages are not lost:

• durable=True (queue survives broker restart)
• delivery_mode=2 (persistent messages)
• Consumers send explicit ACKs to confirm successful processing

If a connection drops before a task is acknowledged, RabbitMQ will safely requeue it for delivery once the worker reconnects.

Once the connection is re-established, the worker continues normal operation. In practice, occasional drops are fine, as long as they remain infrequent and queue depth doesn’t build up.

To end off

That’s all folks, these are some of the key lessons I’ve learned running RabbitMQ + Celery in production. I hope this deep dive has helped you better understand how things work under the hood. If you have more tips, I’d love to hear them in the comments and do reach out!!

Deep dive into RabbitMQ & Python’s Celery: How to optimise your queues? was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

AI-assisted coding is here to stay. Tools like Cursor, V0, and Lovable have dramatically lowered the barrier to entry — building dashboards, pipelines, or entire apps can now be done in a fraction of the time.

I use these tools daily, and they’ve definitely made me faster. But as the codebase gets more complex, the tradeoffs become clear: cryptic bugs, tangled logic, and hours lost debugging code I didn’t truly understand.

AI tools are great — especially for beginners — but they come with a quiet cost. The more you let AI carry the load, the fewer chances you have to sharpen your instincts that come from wrestling with complexity.

Yes, AI will speed up your workflow, but you’ll also skip the formative steps where technical wisdom is earned.

“Vibe coding” — quickly cobbling together code with minimal planning — is great for demos or experiments. But for deeper technical growth or building systems with meaningful complexity, vibe coding isn’t enough. This trending Reddit post sums it up perfectly: left unchecked, vibe coding creates more problems than it solves.

In this piece, I’ll share tips on how to use AI-assisted tools wisely — and why contributing to open source might be the most underrated way to truly level up your technical skills.

My experience vibe coding with Cursor

Like many developers, I switched from VS Code (with GitHub Copilot) to Cursor and am currently subscribed to Cursor’s Pro plan ($20/month).

The feature I rely on most is Cursor’s integrated AI chat, which lets me directly interact with my entire codebase. Its agent can quickly grep through multiple files and even handle images — extremely useful when navigating large, unfamiliar repos. It also spots linter errors and auto-corrects them while directly editing files.

Initially, Cursor dramatically boosted my productivity, especially for simpler tasks. It felt powerful, almost magical. But as things got complex, I noticed some cracks. Cursor would sometimes generate spaghetti code, mix up similarly named files across directories, and occasionally struggle to follow intricate logic flows.

Vibe coding can get you thousands of lines of code in minutes — but without a strong mental model of what you’re building, you risk ending up with bloated, over-engineered systems.

Cursor does a decent job narrowing down the search space when debugging. But letting it make unchecked edits does introduce even more bugs than it solves.

Beyond the usual advice to “write better prompts,” one strategy I’ve found especially helpful is telling Cursor not to make direct edits. (It’s surprisingly obedient about this!)

Instead, I explicitly ask it to suggest changes first in the chat interface. Then, I would review each suggestion, decide which edits made sense, and apply them selectively — either manually or through Cursor. Unlike ChatGPT, Cursor’s biggest strength is its contextual awareness of the entire codebase and it’s ability to parse through lengthy files (over 5k lines of code) by processing them in manageable chunks.

Contributing to open source

So, how do you get technically stronger? Two ways stand out: side projects and open source contributions.

Side projects are great for exploring new tech or diving deep into something you’re passionate or curious about. Wonder how AI agents work or curious about MCP? Just building a simple weekend project teaches you far more than hours of tutorials or documentation. Thanks to open-source, tools and resources are freely accessible, leveling the playing field for everyone.

But solo projects have downsides. It’s easy to lose motivation — many of my own side projects never saw the light of day.

Plus, you can find yourself in an echo chamber: your code works, but you’re not sure if it’s following best practices or industry standards. If you’re early in your career and lack mentorship, how do you know if you’re even on the right track?

This is exactly where open source fills the gap. Open source projects aren’t just for coding wizards; they’re for everyone. Your favourite libraries like Pandas, Matplotlib, TensorFlow, and Keras rely heavily on community involvement.

Why bother contributing?

Open source lets you make a real impact used by thousands of developers — not just toy projects nobody sees. You’ll become proficient with version control (hello, GitHub!), sharpen your skills navigating complex codebases, pick up best practices, and build a network of people who can vouch for you when it matters.

There are career benefits too. It’ll add to your portfolio and personal brand, and you’ll ramp up faster when joining new teams.

But, contribute for the right reasons. If your only motivation is landing a job, DON’T contribute! Open source is not a ticket to get a job — it requires genuine interest and commitment. It shows you’ve a passion to build, and for many startups that begin from open source projects, that’s how they find their first hires.

Picking a open source project that you care about

Starting out can seem daunting. Many popular repos have enormous codebases, potentially outdated documentation, or a bunch of unclear issues. So how do you pick?

First up, pick a project you genuinely care about. This might sound obvious, but it’s crucial — and underrated.

Choose something you actually use, whether at work or in a side project. Jumping into an unfamiliar project with unfamiliar tech is simply overwhelming, and you’ll lose motivation fast.

Personally, I’m both a user and a big fan of PostHog — the product analytics platform built specifically for developers — so I started contributing there. Their docs were comprehensive and well-structured, which made it an awesome place to start. (And no, they didn’t pay me to say this!)

What to contribute?

There are a ton of things you can do. Here’s an approach that I found helpful.

1. Find a feature you need or improve something you use.
   Narrowing down contributions to features you genuinely care about gives clarity and motivation. The best code comes from solving problems you personally face.
2. Set up your local environment.
   Fork the project, clone it locally, and get it running. Understand where logs are and how to test changes. Get a grasp on the project’s basic structure and coding style.
3. Start small and learn by doing
   Many repos tag beginner-friendly issues (like “good-first-issue”). Pick these to start. Understand and replicate the bug; don’t hesitate to comment if you’re stuck. When you open a PR, ensure your changes pass all linting and tests.

Learning to navigate the codebase is essential. You don’t need to read every line — that’s practically impossible. After grasping the high-level structure, dive in. Start small to get comfortable with the build, deployment, and PR review process. Write clear commit messages and PR descriptions. Check recently merged PRs to see successful examples or insightful discussions.

Wrapping up

Contributing to open source takes patience — popular repos are huge, and learning takes time. Becoming a consistent, valuable contributor takes at least a few months, so don’t get discouraged by initial setbacks. If your PR is rejected or you get stuck on a tricky bug, that’s perfectly normal — it’s all part of the learning process.

If you’re new to open source and want to chat, feel free to connect. While I didn’t dive deeply into technical details here (a quick Google or ChatGPT search can guide you there), I hope this gives you the big-picture perspective to get started. Open source has been rewarding for me — and I hope it will be for you too.

See you in the next article :)

How to level up your technical skills in this AI era & contribute to open source was originally published in Women in Technology on Medium, where people are continuing the conversation by highlighting and responding to this story.

Welcome to part 2 of my LLM deep dive. If you’ve not read Part 1, I highly encourage you to check it out first.

Previously, we covered the first two major stages of training an LLM:

1. Pre-training — Learning from massive datasets to form a base model.
2. Supervised fine-tuning (SFT) — Refining the model with curated examples to make it useful.

Now, we’re diving into the next major stage: Reinforcement Learning (RL). While pre-training and SFT are well-established, RL is still evolving but has become a critical part of the training pipeline.

I’ve taken reference from Andrej Karpathy’s widely popular 3.5-hour YouTube. Andrej is a founding member of OpenAI, his insights are gold — you get the idea.

Let’s go 🚀

What’s the purpose of reinforcement learning (RL)?

Humans and LLMs process information differently. What’s intuitive for us — like basic arithmetic — may not be for an LLM, which only sees text as sequences of tokens. Conversely, an LLM can generate expert-level responses on complex topics simply because it has seen enough examples during training.

This difference in cognition makes it challenging for human annotators to provide the “perfect” set of labels that consistently guide an LLM toward the right answer.

RL bridges this gap by allowing the model to learn from its own experience.

Instead of relying solely on explicit labels, the model explores different token sequences and receives feedback — reward signals — on which outputs are most useful. Over time, it learns to align better with human intent.

Intuition behind RL

LLMs are stochastic — meaning their responses aren’t fixed. Even with the same prompt, the output varies because it’s sampled from a probability distribution.

We can harness this randomness by generating thousands or even millions of possible responses in parallel. Think of it as the model exploring different paths — some good, some bad. Our goal is to encourage it to take the better paths more often.

To do this, we train the model on the sequences of tokens that lead to better outcomes. Unlike supervised fine-tuning, where human experts provide labeled data, reinforcement learning allows the model to learn from itself.

The model discovers which responses work best, and after each training step, we update its parameters. Over time, this makes the model more likely to produce high-quality answers when given similar prompts in the future.

But how do we determine which responses are best? And how much RL should we do? The details are tricky, and getting them right is not trivial.

RL is not “new” — It can surpass human expertise (AlphaGo, 2016)

A great example of RL’s power is DeepMind’s AlphaGo, the first AI to defeat a professional Go player and later surpass human-level play.

In the 2016 Nature paper (graph below), when a model was trained purely by SFT (giving the model tons of good examples to imitate from), the model was able to reach human-level performance, but never surpass it.

The dotted line represents Lee Sedol’s performance — the best Go player in the world.

This is because SFT is about replication, not innovation — it doesn’t allow the model to discover new strategies beyond human knowledge.

However, RL enabled AlphaGo to play against itself, refine its strategies, and ultimately exceed human expertise (blue line).

RL represents an exciting frontier in AI — where models can explore strategies beyond human imagination when we train it on a diverse and challenging pool of problems to refine it’s thinking strategies.

RL foundations recap

Let’s quickly recap the key components of a typical RL setup:

• Agent
  The learner or decision maker. It observes the current situation (state), chooses an action, and then updates its behaviour based on the outcome (reward).
• Environment — the external system in which the agent operates.
• State — a snapshot of the environment at a given step t.

At each timestamp, the agent performs an action in the environment that will change the environment’s state to a new one. The agent will also receive feedback indicating how good or bad the action was.

This feedback is called a reward, and is represented in a numerical form. A positive reward encourages that behaviour, and a negative reward discourages it.

By using feedback from different states and actions, the agent gradually learns the optimal strategy to maximise the total reward over time.

Policy

The policy is the agent’s strategy. If the agent follows a good policy, it will consistently make good decisions, leading to higher rewards over many steps.

In mathematical terms, it is a function that determines the probability of different outputs for a given state — (πθ(a|s)).

Value function

An estimate of how good it is to be in a certain state, considering the long term expected reward. For an LLM, the reward might come from human feedback or a reward model.

Actor-Critic architecture

It is a popular RL setup that combines two components:

1. Actor — Learns and updates the policy (πθ), deciding which action to take in each state.
2. Critic — Evaluates the value function (V(s)) to give feedback to the actor on whether its chosen actions are leading to good outcomes.

How it works:

• The actor picks an action based on its current policy.
• The critic evaluates the outcome (reward + next state) and updates its value estimate.
• The critic’s feedback helps the actor refine its policy so that future actions lead to higher rewards.

Putting it all together for LLMs

The state can be the current text (prompt or conversation), and the action can be the next token to generate. A reward model (eg. human feedback), tells the model how good or bad it’s generated text is.

The policy is the model’s strategy for picking the next token, while the value function estimates how beneficial the current text context is, in terms of eventually producing high quality responses.

DeepSeek-R1 (published 22 Jan 2025)

To highlight RL’s importance, let’s explore DeepSeek-R1, a reasoning model achieving top-tier performance while remaining open-source. The paper introduced two models: DeepSeek-R1-Zero and DeepSeek-R1.

• DeepSeek-R1-Zero was trained solely via large-scale RL, skipping supervised fine-tuning (SFT).
• DeepSeek-R1 builds on it, addressing encountered challenges.

Let’s dive into some of these key points.

1. RL algo: Group Relative Policy Optimisation (GRPO)

One key game changing RL algorithm is Group Relative Policy Optimisation (GRPO), a variant of the widely popular Proximal Policy Optimisation (PPO). GRPO was introduced in the DeepSeekMath paper in Feb 2024.

Why GRPO over PPO?

PPO struggles with reasoning tasks due to:

1. Dependency on a critic model.
   PPO needs a separate critic model, effectively doubling memory and compute.
   Training the critic can be complex for nuanced or subjective tasks.
2. High computational cost as RL pipelines demand substantial resources to evaluate and optimise responses.
3. Absolute reward evaluations
   When you rely on an absolute reward — meaning there’s a single standard or metric to judge whether an answer is “good” or “bad” — it can be hard to capture the nuances of open-ended, diverse tasks across different reasoning domains.

How GRPO addressed these challenges:

GRPO eliminates the critic model by using relative evaluation — responses are compared within a group rather than judged by a fixed standard.

Imagine students solving a problem. Instead of a teacher grading them individually, they compare answers, learning from each other. Over time, performance converges toward higher quality.

How does GRPO fit into the whole training process?

GRPO modifies how loss is calculated while keeping other training steps unchanged:

1. Gather data (queries + responses)
   - For LLMs, queries are like questions
   - The old policy (older snapshot of the model) generates several candidate answers for each query
2. Assign rewards — each response in the group is scored (the “reward”).
3. Compute the GRPO loss
   Traditionally, you’ll compute a loss — which shows the deviation between the model prediction and the true label.
   In GRPO, however, you measure:
   a) How likely is the new policy to produce past responses?
   b) Are those responses relatively better or worse?
   c) Apply clipping to prevent extreme updates.
   This yields a scalar loss.
4. Back propagation + gradient descent
   - Back propagation calculates how each parameter contributed to loss
   - Gradient descent updates those parameters to reduce the loss
   - Over many iterations, this gradually shifts the new policy to prefer higher reward responses
5. Update the old policy occasionally to match the new policy.
   This refreshes the baseline for the next round of comparisons.

2. Chain of thought (CoT)

Traditional LLM training follows pre-training → SFT → RL. However, DeepSeek-R1-Zero skipped SFT, allowing the model to directly explore CoT reasoning.

Like humans thinking through a tough question, CoT enables models to break problems into intermediate steps, boosting complex reasoning capabilities. OpenAI’s o1 model also leverages this, as noted in its September 2024 report: o1’s performance improves with more RL (train-time compute) and more reasoning time (test-time compute).

DeepSeek-R1-Zero exhibited reflective tendencies, autonomously refining its reasoning.

A key graph (below) in the paper showed increased thinking during training, leading to longer (more tokens), more detailed and better responses.

Without explicit programming, it began revisiting past reasoning steps, improving accuracy . This highlights chain-of-thought reasoning as an emergent property of RL training.

The model also had an “aha moment” (below) — a fascinating example of how RL can lead to unexpected and sophisticated outcomes.

Note: Unlike DeepSeek-R1, OpenAI does not show full exact reasoning chains of thought in o1 as they are concerned about a distillation risk — where someone comes in and tries to imitate those reasoning traces and recover a lot of the reasoning performance by just imitating. Instead, o1 just summaries of these chains of thoughts.

Reinforcement learning with Human Feedback (RLHF)

For tasks with verifiable outputs (e.g., math problems, factual Q&A), AI responses can be easily evaluated. But what about areas like summarisation or creative writing, where there’s no single “correct” answer?

This is where human feedback comes in — but naïve RL approaches are unscalable.

Let’s look at the naive approach with some arbitrary numbers.

That’s one billion human evaluations needed! This is too costly, slow and unscalable. Hence, a smarter solution is to train an AI “reward model” to learn human preferences, dramatically reducing human effort.

Ranking responses is also easier and more intuitive than absolute scoring.

Upsides of RLHF

• Can be applied to any domain, including creative writing, poetry, summarisation, and other open-ended tasks.
• Ranking outputs is much easier for human labellers than generating creative outputs themselves.

Downsides of RLHF

• The reward model is an approximation — it may not perfectly reflect human preferences.
• RL is good at gaming the reward model — if run for too long, the model might exploit loopholes, generating nonsensical outputs that still get high scores.

Do note that RLHF is not the same as traditional RL.

For empirical, verifiable domains (e.g., math, coding), RL can run indefinitely and discover novel strategies. RLHF, on the other hand, is more like a fine-tuning step to align models with human preferences.

To end off

And that’s a wrap! I hope you enjoyed Part 2 :) If you haven’t already read Part 1 — do check it out here.

Got questions or ideas for what I should cover next? Drop them in the comments — I’d love to hear your thoughts. See you in the next article!

How LLMs work: Reinforcement learning, RLHF, DeepSeek R1, OpenAI o1, AlphaGo was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

With the recent explosion of interest in large language models (LLMs), they often seem almost magical. But let’s demystify them.

I wanted to step back and unpack the fundamentals — breaking down how LLMs are built, trained, and fine-tuned to become the AI systems we interact with today.

This two-part deep dive is something I’ve been meaning to do for a while and was also inspired by Andrej Karpathy’s widely popular 3.5-hour YouTube video, which has racked up 800,000+ views in just 10 days. Andrej is a founding member of OpenAI, his insights are gold— you get the idea.

If you have the time, his video is definitely worth watching. But let’s be real — 3.5 hours is a long watch. So, for all the busy folks who don’t want to miss out, I’ve distilled the key concepts from the first 1.5 hours into this 10-minute read, adding my own breakdowns to help you build a solid intuition.

What you’ll get

Part 1 (this article): Covers the fundamentals of LLMs, including pre-training to post-training, neural networks, hallucinations, and inference.

Part 2: Reinforcement learning with human/AI feedback, investigating o1 models, DeepSeek R1, AlphaGo

Let’s go! I’ll start with looking at how LLMs are being built.

At a high level, there are 2 key phases: pre-training and post-training.

1. Pre-training

Before an LLM can generate text, it must first learn how language works. This happens through pre-training, a highly computationally intensive task.

Step 1: Data collection and preprocessing

The first step in training an LLM is gathering as much high-quality text as possible. The goal is to create a massive and diverse dataset containing a wide range of human knowledge.

One source is Common Crawl, which is a free, open repository of web crawl data containing 250 billion web pages over 18 years. However, raw web data is noisy — containing spam, duplicates and low quality content — so preprocessing is essential.

If you’re interested in preprocessed datasets, FineWeb offers a curated version of Common Crawl, and is made available on hugging face.

Once cleaned, the text corpus is ready for tokenization.

Step 2: Tokenization

Before a neural network can process text, it must be converted into numerical form. This is done through tokenization, where words, subwords, or characters are mapped to unique numerical tokens.

Think of tokens as the building blocks — the fundamental building blocks of all language models. In GPT4, there are 100,277 possible tokens.

A popular tokenizer, Tiktokenizer, allows you to experiment with tokenization and see how text is broken down into tokens. Try entering a sentence, and you’ll see each word or subword assigned a series of numerical IDs.

Step 3: Neural network training

Once the text is tokenized, the neural network learns to predict the next token based on its context. As shown above, the model takes an input sequence of tokens (e.g., “we are cook ing”) and processes it through a giant mathematical expression — which represents the model’s architecture — to predict the next token.

A neural network consists of 2 key parts:

1. Parameters (weights) — the learned numerical values from training.
2. Architecture (mathematical expression) — the structure defining how the input tokens are processed to produce outputs.

Initially, the model’s predictions are random, but as training progresses, it learns to assign probabilities to possible next tokens.

When the correct token (e.g., “food”) is identified, the model adjusts its billions of parameters (weights) through backpropagation — an optimization process that reinforces correct predictions by increasing their probabilities while reducing the likelihood of incorrect ones.

This process is repeated billions of times across massive datasets.

Base model — the output of pre-training

At this stage, the base model has learned:

• How words, phrases and sentences relate to each other
• Statistical patterns in your training data

However, base models are not yet optimised for real-world tasks. You can think of them as an advanced autocomplete system — they predict the next token based on probability, but with limited instruction-following ability.

A base model can sometimes recite training data verbatim and can be used for certain applications through in-context learning, where you guide its responses by providing examples in your prompt. However, to make the model truly useful and reliable, it requires further training.

2. Post training — Making the model useful

Base models are raw and unrefined. To make them helpful, reliable, and safe, they go through post-training, where they are fine-tuned on smaller, specialised datasets.

Because the model is a neural network, it cannot be explicitly programmed like traditional software. Instead, we “program” it implicitly by training it on structured labeled datasets that represent examples of desired interactions.

How post training works

Specialised datasets are created, consisting of structured examples on how the model should respond in different situations.

Some types of post training include:

1. Instruction/conversation fine tuning
   Goal: To teach the model to follow instructions, be task oriented, engage in multi-turn conversations, follow safety guidelines and refuse malicious requests, etc.
   Eg: InstructGPT (2022): OpenAI hired some 40 contractors to create these labelled datasets. These human annotators wrote prompts and provided ideal responses based on safety guidelines. Today, many datasets are generated automatically, with humans reviewing and editing them for quality.
2. Domain specific fine tuning
   Goal: Adapt the model for specialised fields like medicine, law and programming.

Post training also introduces special tokens — symbols that were not used during pre-training — to help the model understand the structure of interactions. These tokens signal where a user’s input starts and ends and where the AI’s response begins, ensuring that the model correctly distinguishes between prompts and replies.

Now, let’s move on to some other key concepts.

Inference — how the model generates new text

Inference can be performed at any stage, even midway through pre-training, to evaluate how well the model has learned.

When given an input sequence of tokens, the model assigns probabilities to all possible next tokens based on patterns it has learned during training.

Instead of always choosing the most likely token, it samples from this probability distribution — similar to flipping a biased coin, where higher-probability tokens are more likely to be selected.

This process repeats iteratively, with each newly generated token becoming part of the input for the next prediction.

Token selection is stochastic and the same input can produce different outputs. Over time, the model generates text that wasn’t explicitly in its training data but follows the same statistical patterns.

Hallucinations — when LLMs generate false info

Why do hallucinations occur?

Hallucinations happen because LLMs do not “know” facts — they simply predict the most statistically likely sequence of words based on their training data.

Early models struggled significantly with hallucinations.

For instance, in the example below, if the training data contains many “Who is…” questions with definitive answers, the model learns that such queries should always have confident responses, even when it lacks the necessary knowledge.

When asked about an unknown person, the model does not default to “I don’t know” because this pattern was not reinforced during training. Instead, it generates its best guess, often leading to fabricated information.

How do you reduce hallucinations?

Method 1: Saying “I don’t know”

Improving factual accuracy requires explicitly training the model to recognise what it does not know — a task that is more complex than it seems.

This is done via self interrogation, a process that helps define the model’s knowledge boundaries.

Self interrogation can be automated using another AI model, which generates questions to probe knowledge gaps. If it produces a false answer, new training examples are added, where the correct response is:
“I’m not sure. Could you provide more context?”

If a model has seen a question many times in training, it will assign a high probability to the correct answer.

If the model has not encountered the question before, it distributes probability more evenly across multiple possible tokens, making the output more randomised. No single token stands out as the most likely choice.

Fine tuning explicitly trains the model to handle low-confidence outputs with predefined responses.

For example, when I asked ChatGPT-4o, “Who is asdja rkjgklfj?”, it correctly responded: “I’m not sure who that is. Could you provide more context?”

Method 2: Doing a web search

A more advanced method is to extend the model’s knowledge beyond its training data by giving it access to external search tools.

At a high level, when a model detects uncertainty, it can trigger a web search. The search results are then inserted into a model’s context window — essentially allowing this new data to be part of it’s working memory. The model references this new information while generating a response.

Vague recollections vs working memory

Generally speaking, LLMs have two types of knowledge access.

1. Vague recollections — the knowledge stored in the model’s parameters from pre-training. This is based on patterns it learned from vast amount of internet data but is not precise nor searchable.
2. Working memory — the information that is available in the model’s context window, which is directly accessible during inference. Any text provided in the prompt acts as a short term memory, allowing the model to recall details while generating responses.

Adding relevant facts within the context window significantly improves response quality.

Knowledge of self

When asked questions like “Who are you?” or “What built you?”, an LLM will generate a statistical best guess based on its training data, unless explicitly programmed to respond accurately.

LLMs do not have true self-awareness, their responses depend on patterns seen during training.

One way to provide the model with a consistent identity is by using a system prompt, which sets predefined instructions about how it should describe itself, its capabilities, and its limitations.

To end off

That’s a wrap for Part 1! I hope this has helped you build intuition on how LLMs work. In Part 2, we’ll dive deeper into reinforcement learning and some of the latest models.

Got questions or ideas for what I should cover next? Drop them in the comments — I’d love to hear your thoughts. See you in Part 2! :)

How LLMs work: Pre-training to post-training, neural networks, hallucinations & inference was originally published in Level Up Coding on Medium, where people are continuing the conversation by highlighting and responding to this story.

Real-time communication is everywhere — live chatbots, data streams, or instant messaging. WebSockets are a powerful enabler of this, but when should you use them? How do they work, and how do they differ from traditional HTTP requests?

This article was inspired by a recent system design interview — “design a real time messaging app” — where I stumbled through some concepts. Now that I’ve dug deeper, I’d like to share what I’ve learned so you can avoid the same mistakes.

In this article, we’ll explore how WebSockets fit into the bigger picture of client‑server communication. We’ll discuss what they do well, where they fall short, and — yes — how to design a real‑time messaging app.

Client-server communication

At its core, client-server communication is the exchange of data between two entities: a client and a server.

The client requests for data, and the server processes these requests and returns a response. These roles are not exclusive — services can act as both a client and a server simultaneously, depending on the context.

Before diving into the details of WebSockets, let’s take a step back and explore the bigger picture of client-server communication methods.

1. Short polling

Short polling is the simplest, most familiar approach.

The client repeatedly sends HTTP requests to the server at regular intervals (e.g., every few seconds) to check for new data. Each request is independent and one-directional (client → server).

This method is easy to set up but can waste resources if the server rarely has fresh data. Use it for less time‑sensitive applications where occasional polling is sufficient.

2. Long polling

Long polling is an improvement over short polling, designed to reduce the number of unnecessary requests. Instead of the server immediately responding to a client request, the server keeps the connection open until new data is available. Once the server has data, it sends the response, and the client immediately establishes a new connection.

Long polling is also stateless and one-directional (client → server).

A typical example is a ride‑hailing app, where the client waits for a match or booking update.

3. Webhooks

Webhooks flip the script by making the server the initiator. The server sends HTTP POST requests to a client-defined endpoint whenever specific events occur.

Each request is independent and does not rely on a persistent connection. Webhooks are also one-directional (server to client).

Webhooks are widely used for asynchronous notifications, especially when integrating with third-party services. For example, payment systems use webhooks to notify clients when the status of a transaction changes.

4. Server-Sent Events (SSE)

SSEs are a native HTTP-based event streaming protocol that allows servers to push real-time updates to clients over a single, persistent connection.

SSE works using the EventSource API, making it simple to implement in modern web applications. It is one-directional (server to client) and ideal for situations where the client only needs to receive updates.

SSE is well-suited for applications like trading platforms or live sports updates, where the server pushes data like stock prices or scores in real time. The client does not need to send data back to the server in these scenarios.

But what about two-way communication?

All the methods above focus on one‑directional flow. For true two‑way, real‑time exchanges, we need a different approach. That’s where WebSockets shine.

Let’s dive in.

How do WebSockets work?

WebSockets enable real-time, bidirectional communication, making them perfect for applications like chat apps, live notifications, and online gaming. Unlike the traditional HTTP request-response model, WebSockets create a persistent connection, where both client and server can send messages independently without waiting for a request.

The connection begins as a regular HTTP request and is upgraded to a WebSocket connection through a handshake.

Once established, it uses a single TCP connection, operating on the same ports as HTTP (80 and 443). Messages sent over WebSockets are small and lightweight, making them efficient for low-latency, high-interactivity use cases.

WebSocket connections follow a specific URI format: ws:// for regular connections and wss:// for secure, encrypted connections.

What’s a handshake?

A handshake is the process of initialising a connection between two systems. For WebSockets, it begins with an HTTP GET request from the client, asking for a protocol upgrade. This ensures compatibility with HTTP infrastructure before transitioning to a persistent WebSocket connection.

1. Client sends a request, with headers that look like:

GET /chat HTTP/1.1
Host: server.example.com
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Key: dGhlIHNhbXBsZSBub25jZQ==
Origin: http://example.com
Sec-WebSocket-Protocol: chat, superchat
Sec-WebSocket-Version: 13

• Upgrade — signals the request to switch the protocol
• Sec-WebSocket-Key — Randomly generated, base64 encoded string used for handshake verification
• Sec-WebSocket-Protocol (optional) — Lists subprotocols the client supports, allowing the server to pick one.

2. Server responds to resquest

If the server supports WebSockets and agrees to the upgrade, it responds with a 101 Switching Protocols status. Example headers:

HTTP/1.1 101 Switching Protocols
Upgrade: websocket
Connection: Upgrade
Sec-WebSocket-Accept: s3pPLMBiTxaQ9kYGzzhZRbK+xOo=
Sec-WebSocket-Protocol: chat

• Sec-WebSocket-Accept — Base64 encoded hash of the client’s Sec-WebSocket-Key and a GUID. This ensures the handshake is secure and valid.

3. Handshake validation

With the 101 Switching Protocols response, the WebSocket connection is successfully established and both client and server can start exchanging messages in real time.

This connection will remain open till it is explicitly closed by either party.

If any code other than 101 is returned, the client has to end the connection and the WebSocket handshake will fail.

Here’s a summary.

WebSocket use cases

We’ve talked about how WebSockets enable real-time, bidirectional communication, but that’s still pretty abstract term. Let’s nail down some real examples.

WebSockets are widely used in real-time collaboration tools and chat applications, such as Excalidraw, Telegram, WhatsApp, Google Docs, Google Maps and the live chat section during a YouTube or TikTok live stream.

Trade offs

1. Having a fallback strategy if connections are terminated

WebSockets don’t automatically recover if the connection is terminated due to network issues, server crashes, or other failures. The client must explicitly detect the disconnection and attempt to re-establish the connection.

Long polling is often used as a backup while a WebSocket connection tries to get reestablished.

2. Not optimised for streaming audio and video data

WebSocket messages are designed for sending small, structured messages. To stream large media data, a technology like WebRTC is better suited for these scenarios.

3. WebSockets are stateful, hence horizontally scaling is not trivial

WebSockets are stateful, meaning the server must maintain an active connection for every client. This makes horizontal scaling more complex compared to stateless HTTP, where any server can handle a client request without maintaining persistent state.

You’ll need an additional layer of pub/sub mechanisms to do this.

Design a real time messaging app

Now let’s see how this is applied in system design. I’ve covered both the simple (unscalable) solution and a horizontally scaled one.

Non-scalable single server app: How do two users chat real time?

1. All users connect via WebSocket to one server. The server holds an in-memory mapping of userID : WebSocket conn 1
2. user1 sends the message over its WebSocket connection to the server.
3. The server writes the message to the MessageDB (persistence first).
4. The server then looks up user2 : WebSocket conn 2 in it’s in memory map. If user2 is online, it delivers the message in real time.
5. If user2 is offline, the server writes to InboxDB (a store of undelivered messages). When user2 returns online, the server fetches all offline messages from InboxDB.

Horizontally scaled system: How do two users chat real time?

A single server can only handle so many concurrent WebSockets. To serve more users, you need to horizontally scale your WebSocket connections.

The key challenge: If user1 is connected to server1 but user2 is connected to server2, how does the system know where to send the message?

Redis can be used as a global data store that maps userID : serverID for active WebSocket sessions. Each server updates Redis when a user connects (goes online) or disconnects (goes offline).

For instance:

• user1 connects to server1.
  server1’s in memory map: user1 : WebSocket connection
  server1 also writes to Redis: user1 : server1
• user2 connects to server2.
  server2’s in memory map: user2 : WebSocket connection
  server2 also writes to Redis: user2 : server2

End to end chat flow: user1 sends a message to user2

1. user1 sends a message through it’s WebSocket on server1.
2. server1 passes the message to a Chat Service.
3. Chat Service first writes the message to MessageDB for persistence.
4. Chat Service then checks Redis to get the online/offline status of user2.
5. If user2 is online, Chat Service publishes the message to a message broker, tagging it with: “user2: server2”.
6. The broker then routes the message to server2.
7. server2 looks up it’s local in memory mapping to find the WebSocket connection of user2 and pushes the message real time over that WebSocket.
8. If user2 is offline (no entry in Redis), Chat Service writes the message to the InboxDB. When user2 returns online, Chat Service will fetch all the undelivered messages.
9. Whenever a new WebSocket connection is opened or closed, the servers update Redis.
10. When a user first loads the app or opens a chat, the Chat Service fetches historical messages (e.g., from the last 10 days) from MessageDB. A cache layer can reduce repeated DB queries.

Some important design considerations:

1. Persistence first
   All messages go to the DB before being delivered. If a push to WebSocket fails, the message is still safe in the DB.
2. Redis
   Stores only active connections to minimize overhead.
   A replica can be added to prevent a single point of failure.
3. Inbox DB helps to handle offline cases cleanly.
4. Chat Service abstraction
   The WebSocket servers handle real‐time connections and routing.
   The Chat Service layer handles HTTP requests and all DB writes.
   This separation of concerns makes it easier to scale or evolve each piece.
5. Ensuring in order delivery of messages
   Typical “real time push” workflows can have network variations, leading to messages arriving out of order.
   Many message brokers also do not guarantee strict ordering.
   To handle this, each message is assigned a timestamp at creation. Even if messages arrive out of order, the client can reorder them based on the timestamp.
6. Load balancers
   L4 Load Balancer (TCP) for sticky WebSocket connections.
   L7 Load Balancer (HTTP) for regular requests (CRUD, login, etc).

Wrapping up

That’s all for now! There’s so much more we could explore, but I hope this gave you a solid starting point. Feel free to drop your questions in the comments below :)

I write regularly on Python, software development and the projects I build, so give me a follow to not miss out. See you in the next article.

Deep Dive into WebSockets and Their Role in Client-Server Communication was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.

When you’re deep in rapid prototyping, it’s tempting to skip clean scoping or reuse common variable names (hello, df!), thinking it will save time. But this can lead to sneaky bugs that break your workflow.

The good news? Writing clean, well-scoped code does not require additional effort once you understand the basic principles.

Let’s break it down.

What is variable scope?

Think of a variable as a container that will store some information. Scope refers to the region of your code where a variable is accessible.

Scope prevents accidental changes by limiting where variables can be read or modified. If every variable was accessible from anywhere, you’ll have to keep track of all of them to avoid overwriting it accidentally.

In Python, scope is defined by the LEGB rule, which stands for: local, enclosing, global and built-in.

Scoping in Python: LEGB rule

Let’s illustrate this with an example.

# Global scope, 7% tax
default_tax = 0.07 

def calculate_invoice(price):
    # Enclosing scope
    discount = 0.10  
    total_after_discount = 0

    def apply_discount():
        nonlocal total_after_discount
        
        # Local scope
        tax = price * default_tax  
        total_after_discount = price - (price * discount)
        return total_after_discount + tax

    final_price = apply_discount()
    return final_price, total_after_discount

# Built-in scope
print("Invoice total:", round(calculate_invoice(100)[0], 2)) 

1. Local scope

Variables inside a function are in the local scope. They can only be accessed within that function.

In the example, tax is a local variable inside apply_discount. It is not accessible outside this function.

2. Enclosing scope

These refer to variables in a function that contains a nested function. These variables are not global but can be accessed by the inner (nested) function. In this example, discount and total_after_discount are variables in the enclosing scope of apply_discount .

The nonlocal keyword:

The nonlocal keyword is used to modify variables in the enclosing scope, not just read them.

For example, suppose you want to update the variable total_after_discount, which is in the enclosing scope of the function. Without nonlocal, if you assign to total_after_discount inside the inner function, Python will treat it as a new local variable, effectively shadowing the outer variable. This can introduce bugs and unexpected behavior.

3. Global scope

Variables that are defined outside all functions and accessible throughout.

The global statement

When you declare a variable as global inside a function, Python treats it as a reference to the variable outside the function. This means that changes to it will affect the variable in the global scope.

With the global keyword, Python will create a new local variable.

x = 10  # Global variable

def modify_global():
    global x  # Declare that x refers to the global variable
    x = 20    # Modify the global variable

modify_global()
print(x)  # Output: 20. If "global" was not declared, this would read 10

4. Built-in scope

Refers to the reserved keywords that Python uses for it’s built-in functions, such as print , def , round and so on. This can be accessed at any level.

Key concept: global vs nonlocal keywords

Both keywords are crucial for modifying variables in different scopes, but they’re used differently.

• global: Used to modify variables in the global scope.
• nonlocal: Used to modify variables in the enclosing (non-global) scope.

Variable shadowing

Variable shadowing happens when a variable in an inner scope hides a variable from an outer scope.

Within the inner scope, all references to the variable will point to the inner variable, not the outer one. This can lead to confusion and unexpected outputs if you’re not careful.

Once execution returns to the outer scope, the inner variable ceases to exist, and any reference to the variable will point back to the outer scope variable.

Here’s a quick example. x is shadowed in each scope, resulting in different outputs depending on the context.

#global scope
x = 10

def outer_function():
    #enclosing scope
    x = 20  

    def inner_function():
        #local scope
        x = 30  
        print(x)  # Outputs 30

    inner_function()
    print(x)  # Outputs 20

outer_function()
print(x)  # Outputs 10

Parameter shadowing

A similar concept to variable shadowing, but this occurs when a local variable redefines or overwrites a parameter passed to a function.

def foo(x):
    x = 5  # Shadows the parameter `x`
    return x

foo(10)  # Output: 5

x is passed as 10. But it is immediately shadowed and overwritten by x=5

Scoping in recursive functions

Each recursive call gets its own execution context, meaning that the local variables and parameters in that call are independent of previous calls.

However, if a variable is modified globally or passed down explicitly as a parameter, the change can influence subsequent recursive calls.

• Local variables: These are defined inside the function and only affect the current recursion level. They do not persist between calls.
• Parameters passed explicitly to the next recursive call retain their values from the previous call, allowing the recursion to accumulate state across levels.
• Global variables: These are shared across all recursion levels. If modified, the change will be visible to all levels of recursion.

Let’s illustrate this with an example.

Example 1: Using a global variable (not recommended)

counter = 0  # Global variable

def count_up(n):
    global counter
    if n > 0:
        counter += 1
        count_up(n - 1)

count_up(5)
print(counter)  # Output: 5

counter is a global variable shared across all recursive calls. It gets incremented at each level of recursion, and its final value (5) is printed after the recursion completes.

Example 2: Using parameters (recommended)

def count_up(n, counter=0):
    if n > 0:
        counter += 1
        return count_up(n - 1, counter)
    return counter

result = count_up(5)
print(result)  # Output: 5

• counter is now a parameter of the function.
• counter is passed from one recursion level to the next, with it’s value updated at each level. The counter is not reinitialised in each call, rather, it’s current state is passed forward to the next recursion level.
• The function is now pure — there are no side effects and it only operates within it’s own scope.
• As the recursive function returns, the counter “bubbles up” to the top level and is returned at the base case.

Best practices

1. Use descriptive variable names

Avoid vague names like df or x. Use descriptive names such as customer_sales_df or sales_records_df for clarity.

2. Use capital letters for constants

This is the standard naming convention for constants in Python. For example, MAX_RETRIES = 5.

3. Avoid global variables as much as possible

Global variables introduces bugs and makes code harder to test and maintain. It’s best to pass variables explicitly between functions.

4. Aim to write pure functions where possible

What’s a pure function?

1. Deterministic: It always produces the same output for the same input. It’s not affected by external states or randomness.
2. Side-effect-free: It does not modify any external variables or states. It operates solely within its local scope.

Using nonlocal or global would make the function impure.

However, if you’re working with a closure, you should use the nonlocal keyword to modify variables in the enclosing (outer) scope, which helps prevent variable shadowing.

A closure occurs when a nested function (inner function) captures and refers to variables from its enclosing function (outer function). This allows the inner function to “remember” the environment in which it was created, including access to variables from the outer function’s scope, even after the outer function has finished executing.

The concept of closures can go really deep, so tell me in the comments if this is something I should dive into in the next article! :)

5. Avoid variable shadowing and parameter shadowing

If you need to refer to an outer variable, avoid reusing its name in an inner scope. Use distinct names to clearly distinguish the variables.

Wrapping up

That’s a wrap! Thanks for sticking with me till the end.

Have you encountered any of these challenges in your own work? Drop your thoughts in the comments below!

I write regularly on Python, software development and the projects I build, so give me a follow to not miss out. See you in the next article :)

Why Variable Scoping Can Make or Break Your Data Science Workflow was originally published in TDS Archive on Medium, where people are continuing the conversation by highlighting and responding to this story.