So, my learning friends, today we will dive a little bit into the topic of tools and MCP. What is it? What is the difference between them? Let’s hope at the end of this short reading, both me and you, will have no doubt about it!

As you have probably heard about RAG by now, which is a system that gives a model a superpower of retrieving information from a knowledge base before answering. Think of it like a student that is in the middle of the exam and he or she, instead of relying only on what they memorized while learning, is allowed to check the textbook before writing down an answer. This is already a big advantage. But what if we gave our student even more superpowers? Not just the possibility to retrieve something from the knowledge base, but to pick up a phone and call someone, check on internet, or actually go and do something? That is exactly the journey we are going on today and what the topic is about. So, let’s go!

Enter Tool Calling

Here is where tool calling (sometimes also called as function calling) comes in. If RAG gives a model the superpower to retrieve, tool calling gives it the superpower to act, or at least to ask something else to act.

So, in general, it works like this: first, you present to the model a set of functions (tools) that the model is allowed to use. Not the code itself, just a description which include the name of the function, what the function does, and what inputs it expects. Something like “get_weather(city): looks up weather information for the CITY”. Important to know: model never actually runs this function itself. What it actually does is only decide, based on the conversation, whether calling one of these functions would help answer the question. If yes, it is telling you which one to call and with which inputs.

So, for example, when you ask “what is the weather in Wrocław right now”, the model does not try to guess the answer from training data (which would be a pretty bad idea, since weather changes and its training data has a cutoff date that is much far back). Instead, it recognizes that this question matches the description of your function, and it replies not with a final answer, but with a structured request to call get_weather() with city=Wrocław. Your code is the one that actually executes that call, gets the real result, and hands it back to the model. Only then does the model turn that raw result into a nice, human-sounding sentence. You can look at the really simplify version of this flow below.

As you can see this is solving some problems RAG could not on its own, which is getting live, current information, calling some external services, doing some specific calculations or doing things like sending email. Amazing, no?

Multi-Turn and Loops: It Is Rarely Just One Call

Tool calling is almost never a single, one-shot event. It is mostly a loop. Think about a question like “what is the weather in Wrocław, and should I bring an umbrella to the meeting tomorrow at 4pm?” To answer that properly, the model might need to call get_weather() first, chech the result, realize it also needs to check your calendar to know where “tomorrow at 4pm” is. So, in order to do that it would need to request a call to a second function, something like, get_calendar_event(), look at that result too, and only then put both pieces together into one answer.

So the actual flow looks something like: the model gets your message, decides a tool call is needed, your app runs the tool and returns the result as a new message, the model reads that result and decides whether it needs yet another function call or maybe this time it is finally ready to answer you. This can repeat several times before you or your user ever see a reply. This hidden conversation is happening between the model and your tools, with you or the your user only seeing the final answer at the end.

This loop is also what turns a simple chatbot into something closer to what people now call an agent: not just a model that generate simple answers, but a model that can plan a small sequence of actions, check results, and adjust before giving you the answer.

So Why Is Not This Enough?

Tool calling on its own is powerful, but there is a catch noone will tell you about upfront: every single tool has to be connected by hand, for every single application, again and again.

Only imagine that you are building three different AI apps: one that needs access to your calendar, one that needs your CRM, and one that needs your file storage. With plain tool calling, for each app, you have to write the code that describes the function, write the code that actually executes it, handle the authentication for that specific service, the errors, and keep it all in sync whenever that service changes its API. Now imagine you have ten tools and five different applications that all want to use some combination of them. Suddenly you are not writing ten integrations, you are writing up to fifty, because every app talks to every tool in its own custom way. This is sometimes called the M×N problem: M applications, N tools, and a spaghetti of custom code connecting each of them.

Nobody wants to reinvent the “how do I call Google Calendar” wheel inside every single project they build. And this is exactly the gap MCP was built to close. So, let go find out how!

Enter MCP: Model Context Protocol

MCP, which is short for Model Context Protocol, does not invent a new superpower for the model. It is not a smarter kind of tool calling, and the model itself does not even really know MCP exists. What MCP changes is how tools get delivered to the model in the first place.

Instead of every application writing its own custom integration to every tool, MCP defines one standard way for a tool (or a whole set of tools) to describe itself and expose itself, so that any MCP-compatible application can plug into it without writing additional custom code.

Think of it a bit like USB-C: before it existed, every device had its own charger and its own cable, and connecting anything to anything else was a small nightmare. USB-C did not make your device smarter, it just gave everyone a shared plug. MCP is that shared plug, but for connecting AI applications to tools, data sources, and services. You can see the same call with the use of MCP. I know, for now it looks much more complicated, but at the end, it is making everything easier. Look at this, note your doubts and all the answers you will find in a seconds.

As you see above, MCP setup usually has two sides: an MCP server, which wraps some capability (say, your calendar, your Slack, your weather function) and exposes it in the standard MCP format, and an MCP client, which is inside your application and knows how to talk to any MCP server. The application connects to the server with the client and asks “what can you do?” gets back a list of available tools, and from there it is just… tool calling, the exact same mechanism we already talked about. The only difference is that everything is coordinated between your app, MCP client and MCP server. You do not need to describe manually each tool and sometimes, you do not need to even write the functions. You can use the MCP servers that were provided by others or you can write your own servers to share some common tools for your company or if you want to share the tools with everyone. For example, you can use the MCP servers that were exposed by service providers like code repos or calendar companies. The question that I had, and probably you have right now, is why use MCP if I can use their API deirectly? Well, you can, but then you need to do all the manual work again, writing tool description and code to connect with this API. The MCP servers has this done already, so why doing it again?

As for your application, the flows stays the same. You still pass the question from user to the model. The model still decides when to call a tool and with what input, the same loop still happens.

So, Tools vs. MCP: Where Is the Actual Line?

This is the part I wanted us to leave with no doubt about, because I mixed these up myself in the beginning. Tool calling is a capability: the ability of a model to recognize that a tool call would help, and to ask for it to be called with the right inputs. It is a behavior that happens at the model level, inside that request-response loop we walked through.

MCP is a protocol: a standardized way of packaging, exposing, and discovering those tools (and other context, like resources or prompts but that’s not important for now) so that connecting a new tool does not mean writing another custom integration code from scratch every time. And, just for clarity, you can have tool calling without MCP, plenty of applications did, and still do, by wiring functions by hand. But you cannot really have MCP without tool calling, because MCP still needs that same request-response mechanism to actually let the model use what it exposes.

So if RAG is the librarian who fetches books, tool calling is that same librarian learning she is also allowed to pick up the phone and call other departments when a book is not enough. And MCP is the moment the whole building agrees on one universal phone system, so the librarian does not need to learn a new dialing code every time she wants to reach a new department.

Quick Glossary, For Future Me (and You)

• Tool calling is the ability of a model to recognize that previously described function can help answer a question, and to request that this function will be called with specific inputs, rather than answering directly from memory alone.
• Agent loop is the repeating cycle of the model requesting a tool call, the application executing it and returning the result, and the model deciding whether it needs another call or is ready to answer.
• MCP (Model Context Protocol) is a protocol for exposing tools (and others) to AI applications, so that connecting a new tool does not require custom code for every pair of app and tool.
• MCP server is MCP side that wraps tools and exposes it in the standard MCP format.
• MCP client is MCP side that is inside the application. It has a knowledge how to talk to any MCP server without needing custom code per server.

Wrapping Up

What started as a way to clear up a little bit the topic of tools and MCP ended into its own walk-through. From the librarian (her again) who can only fetch books, to the librarian who can pick up the phone, to even the moment the whole building finally agreed on some one, shared phone system.

As always, if there is one thing worth remembering above everything else, it is this: tool calling is what the model can do, and MCP is how it get delivered to it without the need for everyone to reinvent the same code over and over again. Everything else that we got to know today, the loops, the servers, the clients, it is just engineering built on top of that one idea.

And, as always, writing about it helped me see the line between them far more clearly than just reading about it ever did.

A while ago I wrote about vector databases the same way I’d explain them to a friend over coffee, not the way a manual would. Writing it down helped me understand it better than reading ever did, so I’m doing it again, this time with the question that shows up right after you’ve built your first RAG and it seems to work: how do I actually know if it’s good?

Because “it works” isn’t really an answer. It worked on the five questions you tried while building it. What about the other five thousand questions your users are going to ask, phrased in ways you never anticipated? This is the story of how I learned to answer that properly: the RAG triad, retrieval metrics, where test samples even come from, and how often you’re supposed to check any of this.

It All Starts With a Question: Where Can a RAG Even Go Wrong?

When I first built a RAG pipeline, it was easy to think of it as one thing: question goes in, answer comes out. Something like a little more advanced chat bot. Retrieval and generation just blurred together into a single black box in my head. It took a bit more experience with these systems to realize that’s not really the right mental model. A RAG system is really two systems stitched together: a retriever, which digs through your knowledge base for the relevant bits, and a generator, the LLM that reads those bits and writes the actual answer. And once you start treating them as two separate, connected systems instead of one, evaluation suddenly makes a lot more sense.

That distinction is the single most useful idea in this whole story, because it means there are two completely different places things can go wrong, and they look identical from the outside. The retriever can hand back the wrong documents entirely, the way a librarian might confidently bring you a cookbook when you asked about tax law. Or the retriever can do everything right, and the generator can still ignore what it was handed and make something up anyway, the way a student given the correct textbook page might still answer from memory instead of actually reading it. If you only look at the final answer, you can’t tell which failure you’re looking at, and you’ll waste weeks rewriting your prompt when the real problem was your chunking strategy the whole time.

So lesson one, for me and for you: evaluate retrieval and generation separately, before you ever evaluate them together.

Enter the RAG Triad

I actually first heard the term “RAG triad” in a job interview, of all places. Someone asked me if I knew what it was, and I had to be honest: “Sorry, I haven’t heard that term before, but here’s how I’d actually check if a RAG system is any good,” and I walked them through my own version of it: first check the retrieval, then check the groundedness, and then check whether the final response actually answers what the person asked for. They listened, then told me that what I’d just described was, in essence, the RAG triad. I wrote the term down the second I finished interview and fell into a reading hole about it. It’s rare that one concept reorganizes so much of what you already half-know into something that finally clicks, but this one did it for me.

Let’s dive into the framing that comes from TruEra’s TruLens project, which introduced the RAG triad: three evaluations, one for each seam in the pipeline, that together give you confidence your app isn’t quietly hallucinating.

• Context relevance asks: is what the retriever found actually relevant to the question? Think of it as checking the ingredients before anyone starts cooking. Any irrelevant chunk that sneaks in has the potential to get folded into a hallucinated answer later.
• Groundedness asks: is the generated answer actually supported by that retrieved context, or did the model wander off and answer from its own training data instead? To check this, the response gets broken into individual claims, and each claim is checked for evidence within the retrieved chunks, the same way a teacher checks an essay’s footnotes against the sources it claims to cite.
• Answer relevance asks the question that sounds too obvious to need checking, and is somehow the one people forget most: does the final answer actually address what the user asked? You can retrieve exactly the right context and stay perfectly faithful to it, and still write a technically-accurate answer to a question nobody asked.

What I liked is that it turns a fuzzy worry, “is my RAG hallucinating?”, into three separately-debuggable problems. Low context relevance means go fix your retrieval: chunking, embedding model, top-k. High context relevance but low groundedness means the model had good material and chose not to use it, so go fix your prompt or your generator.

There’s a satisfying logic buried in the RAG architecture: when context relevance goes up, groundedness tends to follow right along with it. It makes perfect sense once you say it out loud. If the model actually has the right material sitting in front of it, it doesn’t need to reach into its own memory to fill the gaps. The same way you’re far less likely to guess on an open-book exam if the textbook is already open to the exact page with the answer.

RAGAS: A Second Angle on the Same Triad

The RAG triad wasn’t the only place where I ran into the idea of separating retrieval quality from generation quality. A framework called RAGAS (Retrieval Augmented Generation Assessment) approaches the same challenge from a different angle, with its own set of evaluation metrics. It was introduced as a way to evaluate RAG pipelines more systematically, including metrics that don’t always require large amounts of hand-labeled data. Instead, it uses automated evaluation techniques, including LLM-based judges, to estimate things like faithfulness, answer relevance, and retrieval quality.

RAGAS centers on four metrics, and once I sat with them for a bit, I noticed they split cleanly into the same two halves as the triad: two metrics scoring the retriever, two scoring the generator.

On the retrieval side:

• Context precision looks at how much of what got retrieved was actually useful for producing a correct answer, penalizing the noisy, irrelevant chunks riding along for no reason.
• Context recall checks whether the retrieved context contains everything needed to support the answer, and unlike the other three, it usually needs a ground-truth answer to compare against.

On the generation side:

• Faithfulness measures whether every claim in the generated answer can actually be traced back to something in the retrieved context, essentially RAGAS’s own version of groundedness.
• Answer relevancy measures whether the generated answer addresses the question that was actually asked, computed by generating plausible questions the answer could be responding to and checking how closely those match the original question.

One detail I appreciated from a walkthrough on this: older text-generation metrics like BLEU and ROUGE fall short here because they mostly reward surface-level word overlap with a reference answer, not whether a claim is actually backed by evidence, a bit like grading an essay purely on whether it uses the same words as the answer key, regardless of whether any of it is true. That’s exactly the gap RAGAS was built to close.

Line RAGAS up next to the RAG triad and you get: context precision/recall map onto context relevance, faithfulness maps onto groundedness, and answer relevancy maps onto answer relevance. Different framework, different vocabulary, same three seams.

Retrieval: The Old, Reliable Metrics

Going a little bit into the details of retrieval evaluation and how we can actually score it. I know, a little boring theory. Retrieval evaluation isn’t a new problem invented for RAG. It’s the exact same task that’s been sitting behind every search bar for decades, from library catalog systems to Google to an internal company wiki. So instead of inventing something new, the RAG world mostly borrowed metrics that information retrieval researchers already had, and put them to work here too.

These metrics all share a shape: pick a number k (say, the top 5 or top 10 retrieved chunks) and ask how good that particular list is.

• Precision@k: of the k chunks you retrieved, how many were actually relevant? This is like asking “how much of what’s in my grocery bag did I actually mean to buy,” versus how much is filler that snuck in.
• Recall@k: of all the relevant chunks that exist anywhere in your knowledge base, how many did you actually manage to find in your top k? This is closer to “of everything on my shopping list, how much did I actually bring home.”
• MRR (Mean Reciprocal Rank): instead of just “did we find it,” this asks “how fast did we find it.” It looks at the position of the first relevant result and averages the reciprocal of that rank across all your test queries. A relevant chunk sitting at position 1 scores far better than the same chunk sitting at position 5, even though a plain recall check would treat them the same.
• NDCG@k (Normalized Discounted Cumulative Gain): the most complete metric of the group, because it accounts for both whether relevant chunks were retrieved and how highly they were ranked. Unlike Recall@k, NDCG rewards placing the most relevant results near the top of the list, and it can also incorporate graded relevance scores when they’re available. It’s one of several metrics used across the MTEB benchmark, but it’s the primary metric for the Retrieval leaderboard (reported as nDCG@10). So if you’ve compared embedding models on MTEB’s retrieval tasks, that’s the score that has likely been driving those rankings.

We can differentiate them into two groups, rank-aware versus not rank-aware. Precision and recall only care whether relevant chunks showed up somewhere in your top k, order doesn’t matter to them at all. MRR and NDCG care a lot about order, and in a RAG system, order genuinely matters, because chunks near the top of the context window tend to have far more influence on the generated answer than chunks quietly buried at the bottom.

To compute any of this, you need a small ground-truth set: a list of queries, each paired with the chunk (or chunks) that should have been retrieved for it. This has nothing to do with generation quality at all, it’s a pure search-quality check, and there are Python libraries (as for everything in Python) that exist specifically to compute all of it efficiently, including statistical significance testing across different retrieval configurations.

Where Do Test Samples Even Come From?

Metrics are great, but a metric needs something to measure against : a set of questions with known-good answers and known-good source chunks. Where does that come from? Here I will talk about three main ways to get there, and they trade off cost against realism.

Hand-labeled golden datasets. A human, ideally someone who actually knows the domain, writes real questions and marks which chunks should answer them. This is the highest-quality option (for now), but it’s expensive, since every question costs real human time, and it doesn’t scale well if your knowledge base keeps changing.

Synthetic test set generation. Instead of a human writing questions, you point an LLM at your own documents and let it generate question/context/answer triples automatically. The RAGAS TestsetGenerator is a well-known example: it builds an internal knowledge graph from your documents, then deliberately generates a mix of question types, single-hop, multi-hop, reasoning, conditional, so your test set doesn’t collapse into the same easy question asked over and over.

Production logs. Once your app is live, the most honest test cases are the ones your actual users typed, typos, weird phrasing, and all. The idea is to increasingly mine hard or failing cases straight out of production traces and “promote” them into a permanent regression set, so every real failure your system has ever produced becomes a permanent guard against it happening again.

Which one to choose? I would say, mix all three: synthetic datasets for fast, broad coverage, a small hand-labeled golden set for the cases that matter most, and a growing collection of real production failures. A common way to think about it is that the auto-generated dataset is silver which is good enough to help you iterate quickly, while the carefully human-verified one is gold, the benchmark you trust before making important decisions.

One last thing to keep in mind: your RAG system should be able to answer new questions (generalize), not just the ones in your evaluation set. If you keep tweaking it to do better on the same benchmark, you might end up making it great at those exact questions but worse at handling new ones. So use your evaluation dataset as a guide, not something to optimize for at all costs.

Generation: How Do You Even Grade an Answer?

Retrieval has it comparatively easy: you can check whether a chunk matches a query with a ground-truth list and some arithmetic. Generation doesn’t offer that luxury. There’s no fixed “correct” sentence a good answer has to match word for word, which is exactly why older metrics like BLEU and ROUGE, built for scoring things like machine translation against a single reference sentence, don’t hold up well here. So how do you grade something as open-ended as a written answer, at scale, without hiring a small army of human reviewers?

The answer that I saw the most is LLM-as-a-judge: you have one LLM score the output of another against criteria you define, things like faithfulness, relevance, helpfulness, or correctness. No annotation team, no waiting weeks for human review, just a prompt and an API call. It’s become the default way to evaluate LLM applications at scale, largely because it turns out an LLM judge agrees with human reviewers roughly as often as two human reviewers agree with each other.

There are two main ways to set this up. In single-output scoring, the judge looks at one answer and rates it against a rubric, this is how most of the groundedness and answer-relevance checks we’ve already talked about actually work under the hood. Here you can also optimize this, extend and use some techniques to get even better results. In the second way, pairwise scoring, the judge is shown two candidate answers for the same question and asked to pick the better one, which tends to be an easier, more reliable judgment for a model to make than assigning an absolute score out of ten.

One thing worth remembering is that LLM judges aren’t neutral referees straight out of the box. They carry real, well-documented biases, things like position bias, verbosity bias, or self-preference bias. I won’t go into the details of each here, that might be its own small write-up one day when I’m refreshing my memory on this topic again. For now, the only thing worth knowing is that these biases exist, and that the way people deal with them is through calibration, tuning the judge so its scores actually line up with what we expected.

It’s worth remembering that LLM-as-a-judge isn’t the only tool in this kit, just the most popular one right now. Some other may include:

Classic n-gram metrics, like BLEU and ROUGE, still show up here and there, even though I mentioned earlier why they struggle with RAG: they count word overlap with a reference answer, so a factually correct answer phrased differently from the reference gets penalized, while an answer that reuses the reference’s words but makes no actual sense can score deceptively well.

Embedding-based metrics, like BERTScore, are a step up, instead of counting exact word matches, they compare the meaning (similarity) of sentences in vector space. That’s no longer blind text matching, but they still need a ready-made reference answer to compare against, so they don’t help in the very common RAG situation where no reference answer exists at all.

Human evaluation remains the benchmark everything else gets compared to, including LLM-as-a-judge itself, as we saw with calibration. It’s the most trustworthy option (for now), but also the slowest and most expensive, so in practice it ends up reserved for small, carefully chosen golden datasets rather than continuous, everyday checking.

Rule-based and programmatic checks earn their place wherever an answer has a rigid structure, checking whether a response actually contains a phone number, a date in the right format, or stays under a length limit. Cheap, fast, and perfectly deterministic, but they can’t judge anything more subtle than format.

In practice, we don’t need to choose only one, these methods complement each other: rules catch obvious formatting errors for free, LLM-as-a-judge scales up to subtler judgments like faithfulness or relevance, and a small, carefully calibrated sample of human-reviewed answers keeps the whole system honest.

Putting It Into Development: Evaluation as Part of CI/CD

Here’s one thing for me, and you, to consider: evaluation isn’t a one-off report you generate the night before a demo. It’s meant to live inside your development workflow, the same way unit tests live inside any other software project.

The pattern I keep seeing across is to write your evaluation dataset as questions and reference answers, not the outputs from your RAG system. That way, every time something changes, you can run your real RAG pipeline against the same questions, compare the fresh results against your evaluation thresholds, and quickly spot regressions. Treat it like unit testing for your RAG system: evaluate retrieval and generation separately, define clear quality thresholds in your CI/CD pipeline, and fail the build if a change makes either one worse. It’s much better to catch a regression during development than have an annoyed user discover it weeks later.

And for that I have one practical idea to consider: tiering your test suite instead of running everything, everywhere, all the time.

• A small, fast smoke test subset runs on every pull request or commit, catching obvious regressions within seconds.
• A full nightly suite runs the complete evaluation set, including slower and more expensive LLM-as-judge metrics that would be too costly or slow to run on every commit.

The other important (last one here, promise) piece is making evaluation results easy to debug. Store the scores together with the exact configuration that produced them: chunk size, top-k, embedding model, generator model, prompt template, and any other relevant settings. This turns vague observations like “our RAG got worse” into something actionable, such as “our RAG got worse after increasing chunk size from 512 to 1024.” Instead of just knowing that quality dropped, you can start understanding why.

How Often Should You Actually Check?

This was my last lingering question, and it turns out the honest answer is as everything: it depends. It depends on whether you’re checking offline or online, because those two run on completely different clocks, but also, it depends on your specific use case. So now, a little about types of evaluations:

Offline evaluation is everything we’ve talked about so far: running your evaluation dataset against your pipeline in a controlled setting. This happens whenever you make a change, a fast subset on every pull request, the full suite on every merge or every night, and always before shipping anything to chunking, prompts, embedding models, or the underlying LLM. This is regression testing, and its entire job is to catch a quality drop before it reaches a real user.

Online evaluation is a different thing entirely: continuous monitoring of live traffic, running lighter versions of the same evaluation metrics (usually reference-free, since real user questions don’t come with a conveniently attached correct answer) against a sample of production interactions in the background. This is how you catch issues that offline testing cannot fully capture: changes in the types of questions users ask, a knowledge base that has become outdated, or an embedding model that quietly changed behavior after a vendor update. A useful practice is to cluster failures across live traffic over time, so recurring patterns surface automatically instead of getting buried in millions of individual logs. The most important failure cases can then be added back into your regression dataset, creating a continuous improvement loop.

So “how often” isn’t really one number, it’s more like a schedule with different frequencies stacked on top of each other:

• On every pull request, a small fast subset.
• On every merge or nightly, the full offline suite.
• Continuously, at some sampled rate, for production monitoring.
• Whenever the corpus itself changes, since new or edited documents can silently shift retrieval and generation quality even if you didn’t touch a single line of code.

But where do you start if you’re not doing any of this yet? Start small and build your evaluation process as your system grows. Begin with a set of fifty to a hundred hand-checked golden cases, add a faithfulness threshold to your CI checks, and log precision@k for every run. It won’t be perfect or sophisticated at first, but it will already give you something valuable: the ability to catch regressions instead of shipping changes blindly.

Quick Glossary, For Future Me (and You)

• Retriever / Generator: the two halves of a RAG pipeline. The retriever finds relevant chunks, the generator writes the answer from them. Always evaluate separately first.
• RAG triad: three-part evaluation, context relevance, groundedness, and answer relevance, one check per seam of the pipeline.
• Context relevance: is the retrieved context actually relevant to the query? The ingredient check before anyone starts cooking.
• Groundedness (faithfulness): is the generated answer actually supported by the retrieved context, or did the model wing it?
• Answer relevance: does the final answer actually address the question that was asked?
• Precision@k / Recall@k: classic, order-unaware retrieval metrics: how much of what you retrieved was relevant, and how much of what was relevant got retrieved.
• MRR (Mean Reciprocal Rank): rewards finding a relevant result early, based on the rank of the first relevant chunk.
• NDCG@k: a graded, rank-aware metric rewarding relevant results that show up both often and near the top.
• RAGAS: an evaluation framework built specifically for RAG, with four core metrics: context precision, context recall, faithfulness, and answer relevancy.
• Golden dataset: a hand-labeled, high-trust evaluation set of questions paired with correct chunks and answers.
• Synthetic test set: an evaluation set generated automatically by an LLM from your own documents, good for broad, fast coverage.
• LLM-as-a-judge: using a carefully prompted LLM to score another model’s output against a rubric, the mechanism behind most RAGAS and RAG triad metrics.
• Offline evaluation: running your evaluation dataset against your pipeline in a controlled setting, usually wired into CI/CD as regression testing.
• Online evaluation: continuous monitoring of live production traffic, catching drift and failure patterns offline tests can’t see.

Wrapping Up

What started as “how do I know my RAG app is actually good” turned into a much longer walk than expected: separating retrieval from generation, meeting the RAG triad in, of all places, a job interview, borrowing decades-old information retrieval metrics, learning where test samples actually come from, and finally realizing evaluation isn’t one event but two clocks ticking at once, one tied to your commits, one tied to your live traffic.

If there’s one thing worth keeping above everything else, it’s this: a RAG system fails in one of three specific places, and the only way to know which one is to measure each of them on purpose instead of squinting at the final answer and guessing. Everything else, the RAG triad, RAGAS, precision@k, CI/CD gates, is just the engineering built on top of that one idea.

And, once again, writing it out helped far more than reading ever did. Maybe I should stop being surprised at this point.

A while ago I read something that started to be important for me in the learning process: you never really learn something until you try to explain it to someone else. Teaching is the best form of learning. So today, instead of just re-reading my old notes on vector databases, I decided to write them down as a story, in a way I’d explain it to a friend over coffee, not the way a textbook would.

If you’ve ever wondered what people mean when they say “we store our data as embeddings” or “we’re using a vector database for our AI app,” this is for you. No jargon walls, just the journey.

It All Starts With a Question: How Does a Computer Understand “Meaning”?

Computers are great at comparing exact things. Is “apple” equal to “apple”? Sure, easy. But is “apple” similar to “fruit”? Or is “king” related to “queen” the same way “man” is related to “woman”? A traditional computer has no idea. It just sees strings of letters.

This is the problem embeddings were invented to solve.

What Are Embeddings, Really?

Imagine you had to describe every word, image, or sound as a list of numbers, being more exact, a set of coordinates in space. Not a random list, but one where similar things end up physically close to each other in that space, and unrelated things end up far apart.

That’s exactly what an embedding is: a numerical representation (called a vector) of a piece of data (which can be almost everything: a text, an image, audio, even a user’s behavior) placed into a high-dimensional space where distance and direction actually carry meaning.

Picture a simple map. On a 2D map, Paris and London are close because they’re both in Europe. Tokyo is far away. Now imagine that instead of just two dimensions (latitude and longitude), you had hundreds or even thousands of dimensions, each capturing some subtle aspect of meaning, for example, tone, topic, sentiment, style. That’s essentially what an embedding model does, except it does it for words, sentences, or entire documents.

So when someone says “the embedding for the word dog,” they mean: a list of numbers, maybe 384, maybe 1536 of them, that encodes what “dog” means in relation to everything else the model has learned.

Where Do These Numbers Come From? Embedding Models

So, where are the numbers from? They’re generated by a model trained specifically for this job. These are called embedding models, and they’re a bit different from the chat-style models most people are familiar with today. Their only purpose is to take an input and output a vector.

Some well-known families of embedding models:

• OpenAI’s text-embedding models: widely used for turning text into vectors for search and retrieval.
• Sentence-BERT (SBERT): an open-source model built specifically to produce meaningful sentence-level embeddings.
• Cohere embed models: another popular commercial option, often used in enterprise search.
• CLIP: instead of just text, it can embed both images and text into the same space, so you can search for images using a text description, or vice versa.

The important thing to remember: an embedding is only meaningful in the context of the model that created it. You can’t mix vectors from two different embedding models and expect them to make sense together because they’re different maps of different worlds.

So What Makes a Vector Database Different From a Normal One?

A traditional database, which you can think as of a classic SQL table, is built to answer exact-match or range questions:

“Give me all users where age > 30” or “find the row where id = 42.”

It’s precise, structured, and predictable.

A vector database is built to answer a completely different kind of question: “Give me the things that are most similar to this one.” There’s no exact match, no equality check. It is just closeness in that high-dimensional space we talked about.

Under the hood, a vector database stores those long lists of numbers (the embeddings) alongside your original data, and it’s optimized to search through millions or billions of these vectors incredibly fast, without having to compare your query to every single one individually (which would be painfully slow at scale).

That’s really the whole shift in mindset: normal databases retrieve by exact matching, vector databases retrieve by meaning.

Every Type of Data Has Its Own Model

Here’s something that took me a moment to fully appreciate: there isn’t one universal embedding model for everything (well, there are some multimodal ones, but let’s forget about them for a second). Text, images, audio, and even structured behavioral data each need their own specialized model, trained on that type of data, to produce vectors that actually capture meaningful similarity.

• Text gets embedded by language-based models (OpenAI embeddings, SBERT, Cohere).
• Images get embedded by vision models, often convolutional or transformer-based (CLIP, for instance, doing both text and images).
• Audio has its own embedding models trained on spectrograms or waveforms.
• Even recommendation systems use embeddings, representing a user or a product as a vector based on behavior, so that “similar users” or “similar products” can be found the same way.

The database itself doesn’t care what the vector represents. It purpose is to just store numbers and to find nearby numbers. The magic of what those numbers mean happens entirely in the embedding model you chose. And this is why, I would say, take some time to select this model if you have a chance to.

Stepping Back: How Does This Actually Work, Start to Finish?

Before we get into how the searching part works, I realized I was skipping over something important: how does data even get into a vector database in the first place, and what actually happens the moment you hit “search”? These are two separate journeys, and once I saw them side by side, the whole system finally made sense as a system, not just a pile of separate concepts.

Journey One: Getting Your Data In (Ingestion)

Picture a librarian who’s just received a truck full of new books. Before shelving anything, she doesn’t just read the title, she is starting w whole process: she skims each book, forms an understanding of what it’s about, and files it near other books with a similar feel. That skimming-and-understanding step is exactly what the embedding model does for your data.

Here’s the walk, step by step: you start with raw data, like a document, a product description, a photo, whatever you’re working with. That raw data gets fed into an embedding model, the same kind we talked about earlier (SBERT, OpenAI’s embedding models, CLIP, and so on). The model reads it and outputs a vector which is, as you already probably know, that long list of numbers capturing its meaning. Finally, that vector gets stored in the database, sitting right alongside the original data it came from, so that when you later find the vector, you can also retrieve the actual document, image, or product behind it.

This whole journey usually happens once, in bulk, ahead of time. This process is often called indexing. You’re not doing this live while someone waits for search results; you’re pre-processing your entire dataset so it’s ready to be searched instantly later.

Journey Two: What Happens the Moment You Search (Query Time)

Now here’s the part that clicked for me the hardest: querying isn’t a fundamentally different process: it’s the same librarian trick, just run in reverse and in real time.

When you type a search query, that query text doesn’t get compared to your stored documents directly. It first has to go through the exact same embedding model that was used during ingestion. This matching is non-negotiable, because a query embedded by a different model would land in a completely different, incompatible meaning-space. Once your query becomes a vector, the database compares it against all the vectors it already has stored, using one of the similarity-search techniques we’re about to cover, and returns the nearest ones by ranking them from most to least similar.

So the whole system really boils down to two mirrored journeys: data goes in through the embedding model once, and every query goes through that same door before it’s allowed to search. Once I saw it laid out like this, similarity search stopped feeling like a mysterious black box and started feeling like simple geometry. Just: “which stored point is closest to my new point?”

Now the Fun Part: Similarity Search

Once everything is turned into vectors, the core operation of a vector database is similarity search. The purpose of this is to find vectors closest to your query vector. This whole approach, so, searching by meaning instead of by exact words, is what people mean when they say vector search: you’re not looking for a matching string anywhere, you’re looking for the nearest point in that meaning-space we talked about earlier.

But “closest” needs a definition, and there are a few common ways to measure it:

• Cosine similarity: measures the angle between two vectors rather than their raw distance. It’s popular for text because it focuses on direction (meaning) rather than magnitude.
• Euclidean distance (L2): the straight-line distance between two points, like measuring with a ruler. Common in image and general-purpose embeddings.
• Dot product: a simpler calculation, often used when vectors are already normalized, and very fast computationally.

There’s also a distinction in how the search is performed:

• Exact k-Nearest Neighbors (kNN): compares your query against every single vector to guarantee the truly closest matches. Accurate, but slow at large scale.
• Approximate Nearest Neighbors (ANN): uses clever indexing structures (like HNSW: Hierarchical Navigable Small World graphs) to find very close matches, almost as good as exact search, but dramatically faster. This is what makes vector databases usable at real-world scale, with millions of records.

Here’s where the story gets interesting, though, because pure vector search, for all its magic, isn’t perfect on its own.

The Problem With “Only Meaning”

Imagine you run an online store, and someone searches for “waterproof jacket for hiking in the rain.” A good embedding model will understand the intent beautifully: it’ll surface jackets described as “rain-resistant outerwear for the trail,” even if those exact words never appear. That’s the whole point of vector search: it reads between the lines.

But now imagine a second customer searches for the exact product code “WPJ-2024-BLK.” A vector search might get confused here. Product codes, serial numbers, and exact names don’t really have “meaning” in the way sentences do. In other words, the embedding model has no real concept of what “WPJ-2024-BLK” is, so it can’t reliably place it near the right product in that meaning-space. This is exactly the kind of query where keyword search, which is the traditional, decades-old technique that scans for exact or near-exact word matches rather than meaning (the same kind of search that’s powered search engines and “Ctrl+F” long before embeddings existed) is unbeatable. It just looks for the string itself and finds it instantly, no understanding required.

So you end up with two very different strengths: vector search is great at understanding, keyword search is great at precision. Neither one alone is enough for a real product. That tension is exactly what hybrid search was built to resolve.

How Hybrid Search Actually Works

Hybrid search doesn’t try to blend the two techniques into one clever new algorithm. It’s much simpler and more elegant than that. It just runs both searches, completely independently, at the same time, and then merges the results afterward.

Think of it like asking two very different experts to judge the same competition. One judge only cares about meaning (as our vector search) they read every entry and rank them by how well they capture the spirit of what you asked for. The other judge only cares about precision (keyword search) they scan for exact keyword matches, spelling, codes, names. Neither judge talks to the other while scoring. They each hand back their own independent ranked list.

The tricky part is: how do you combine two separate rankings into one final list, especially when the underlying scores aren’t even measured on the same scale? A vector search score (say, a cosine similarity of 0.87) and a keyword search score (like a BM25 relevance score of 14.2) can’t just be added together because they’re not speaking the same language.

This is where a technique called Reciprocal Rank Fusion (RRF) comes in, and it’s a genuinely clever trick. Instead of comparing the raw scores from each judge, RRF only looks at the position each result landed in on each list. A result that came 1st in the vector ranking and 3rd in the keyword ranking gets combined differently than one that came 20th and 1st. Results that rank consistently well across both judges naturally rise to the top of the final, merged list, even if neither judge alone gave them the single highest score. It’s a fair, scale-independent way to combine two very different opinions into one confident answer.

The beautiful part is that this all happens in a single request. You send one query, the system quietly runs the vector search and the keyword search in parallel behind the scenes, applies RRF, and hands you back one unified, ranked list. So, from the outside, it just looks like one really smart search.

Quick Glossary, For Future Me (and You)

Since I keep coming back to my own notes and forgetting which term means what, I figured I’d leave myself a cheat sheet right here at the bottom. These are in the order they showed up in the story above.

• Embedding: a list of numbers (a vector) that represents the meaning of a piece of data, such as, text, an image, audio, anything. They are placed so that similar things end up close together.
• Embedding model: the model whose only job is turning raw data into an embedding. Different data types need different embedding models (text, image, audio, behavior). There are also multimodal embedding models that can handle different input types.
• Vector database: a database built to store embeddings and retrieve them by closeness, not by exact match.
• Vector search: the act of searching a vector database: turning your query into a vector too, then finding the stored vectors closest to it. This is search by meaning.
• Keyword search: the older, traditional style of search that scans for exact or near-exact word matches, the way search engines worked long before embeddings existed. Great for things like product codes, names, and exact phrases.
• Similarity search: another name for the core retrieval operation in a vector database: finding the nearest vectors to a given query vector. Often used interchangeably with “vector search.”
• Cosine similarity / Euclidean distance / dot product: the different mathematical rulers used to measure “how close” two vectors are.
• kNN (k-Nearest Neighbors): an exact similarity search that checks every stored vector to guarantee it found the truly closest ones. Accurate but slow at scale.
• ANN (Approximate Nearest Neighbors) / HNSW: a much faster similarity search that checks only a smart shortcut path instead of everyone, trading a tiny bit of accuracy for a lot of speed.
• Hybrid search: running vector search and keyword search in parallel on the same query, then merging both ranked lists into one.
• RRF (Reciprocal Rank Fusion): the technique that merges two separate rankings into one final list, based on each result’s position in each list rather than its raw score.

Wrapping Up

What started as “let me refresh my notes” turned into a full walk-through. From the basic idea of turning meaning into numbers, to picking the right embedding model for your data type, to seeing how data flows in and queries flow through, finally, to understanding similarity search.

If there’s one thing worth remembering above everything else, it’s this: embeddings are how we translate meaning into math, and vector databases are how we search that math at scale. Everything else, cosine similarity, HNSW, hybrid search, is just the engineering built on top of that one idea.

And honestly, writing this out helped me understand it better than reading ever did. Maybe that’s the real takeaway.

When did every startup pitch start sounding like it was written by the same person?

I work in the startup world every day. I see the products, the decks, the roadmaps, the long emails that answer every question except the one you actually asked. And lately, something feels off. Not broken but just… flat. Like someone turned down the contrast on everything. The sharp edges are gone. The ideas, the energy, the “this might be crazy but hear me out” moments, they’re getting rarer.

What replaced them is polished. Grammatically perfect. Structured beautifully. And completely forgettable.

I don’t think this is a coincidence. I think we’re watching AI quietly do something nobody planned for: homogenize the startup world at exactly the moment it needs to be most differentiated. And beyond the aesthetics of it, beyond the boring pitch decks and the copy-paste product visions, there’s a human cost that doesn’t show up in any report. People stop caring. They stop pushing. And why even try when the response to something ambitious is: “I could have done this in seconds with AI. This is too time-consuming.” The message, whether intended or not, is clear: volume matters more than depth. Speed matters more than substance. Produce more. Mean less.

That’s not just a creativity problem. It’s a survival problem.

Everyone Looks the Same. And Investors Notice.

Here’s something that doesn’t appear in any shutdown report: the quiet exhaustion of investors who have sat through hundreds of pitches that feel like variations of the same deck.

Venture capitalists are openly talking about “thin AI wrappers” which are products that are essentially a polished interface layered on top of a foundation model, with no proprietary data, no real moat. Will Richardson, Managing Partner of Giant Leap, described it well: “I like to think LLMs are like the greatest cover band in history. They can play anything, flawlessly, on demand. But every song belongs to someone else… There is no risk and therefore no edge.” One CIO interviewed by MIT researchers was equally blunt: “We’ve seen dozens of demos this year. Maybe one or two are genuinely useful. The rest are wrappers or science projects.”

The research backs this up. A 2024 study by Anderson, Shah and Kreminski found something counterintuitive: while AI helps individuals generate more ideas, it makes those ideas more similar across users. Everyone iterates on the same handful of concepts, just with different brand colors and a slightly different onboarding flow.

Why AI Makes This Worse: The Mechanism Nobody Talks About

The genericity problem isn’t just about pitch decks. It runs deeper into the daily work of building a company. And understanding how it gets there matters, because it’s not obvious from the outside.

Start with what researchers call “cognitive offloading.” When people delegate thinking to AI everything, not just writing or formatting, but actual reasoning, prioritization, judgment and they gradually stop exercising those muscles themselves. A 2025 MIT Media Lab study tracked brain activity across groups writing with and without AI assistance. The result was striking: regular AI users showed the weakest neural connectivity, and most of participants couldn’t recall or cite their own AI-assisted essays shortly after writing them. The researchers coined a term for the accumulated effect: cognitive debt.

At the individual level, this is concerning. At the team level, it’s dangerous.

Harvard Business Review, in collaboration with BetterUp Labs and Stanford, identified a related phenomenon they called “workslop” which is AI-generated content that looks like work but lacks the substance to actually move anything forward. Their survey of 1,150 workers found that 40% had received workslop in the past month. Each incident cost nearly two hours to untangle. But the most alarming finding wasn’t the time lost, it was the interpersonal damage. Around half of recipients said they viewed the sender as less creative and less capable. 42% trusted them less. 32% were less willing to work with them again.

In a startup, where trust and speed of collaboration are everything, this is corrosive.

Then there’s the code. GitClear’s 2025 analysis of hundreds of millions of lines found a dramatic increase in duplicated code blocks, while refactoring, the work that keeps a codebase healthy, dropped sharply over the same period. Google’s DORA 2024 report found that higher AI adoption correlated with lower deployment stability even as individual productivity metrics improved.

This is what “false velocity” looks like in practice. The team is shipping. The commits are flowing. The emails are long and well-formatted. But the underlying product is accumulating debt: technical, cognitive, and relational, faster than anyone realizes. By the time it surfaces, it’s usually in a due diligence call, a churn spike, or a Series A that never materializes.

The cruel irony is that AI was supposed to give startups an edge. For some, it does. But used without discipline, it creates something far more insidious than laziness: the appearance of momentum without the substance of it.

What Investors Actually See

Investors don’t talk about cognitive debt in term sheets. But they feel its effects.

The shift in how experienced VCs evaluate early-stage companies has been quiet but consistent. A few years ago, a polished deck and a confident narrative were strong signals. Today, polish is table stakes, and increasingly, a warning sign. When everything looks good on the surface, the question becomes: what’s underneath?

Some investors have been explicit about what they’re looking for and what they’re avoiding. Abdul Abdirahman of F-Prime noted that generic vertical software “without proprietary data moats” is no longer popular. Igor Ryabenkiy of AltaIR Capital went further: “If your differentiation lives mostly in UI and automation, that’s no longer enough. The barrier to entry has dropped, which makes building a real moat much harder.” Aaron Holiday of 645 Ventures added that what bores investors today are startups building thin workflow layers, generic horizontal tools, and surface-level analytics “basically, anything an AI agent can now do.”

There’s also a subtler shift happening in how investors read teams, not just products. Workslop doesn’t stay internal. It leaks into investor updates, into responses to due diligence questions, into the quality of thinking visible in a founding team’s written communication. An investor who receives a beautifully formatted update that doesn’t actually answer their question, sound familiar?, is getting data. Just not the kind the founder intended to share.

None of this means investors have become hostile to AI. The opposite. AI-native companies that demonstrate genuine differentiation, proprietary insight, or deep workflow integration are getting funded. The bar hasn’t risen against AI. It’s risen against the appearance of AI as a strategy.

The Irony Nobody Planned For

Here’s the paradox at the center of all this: AI was supposed to be the great equalizer. The tool that would let a two-person team compete with a fifty-person team. The thing that would finally give scrappy founders the leverage they needed to punch above their weight.

And in some ways, it delivered. Tasks that used to take days take hours. Founders who couldn’t afford a full marketing team can now ship content. Developers who worked alone can now move faster than small agencies.

But the equalizer cuts both ways. When everyone has access to the same tools, the same models, the same default outputs, the advantage disappears. What’s left is a market where the floor has risen but the ceiling hasn’t. Everyone is more capable of producing something adequate. Almost nobody is producing something surprising.

The startups that are actually winning with AI aren’t using it to replace thinking. They’re using it to go deeper into the thinking they were already doing. They have a point of view that existed before the prompt. They use AI to move faster toward something specific. Not to generate the something in the first place.

That distinction sounds simple. In practice, under pressure, with deadlines and investors and a team waiting for direction, it’s remarkably hard to maintain.

And that’s exactly why it matters. The startups that figure it out don’t just build better products. They build teams that still know how to think, collaborate, and surprise people. In a market full of cover bands, that’s what makes you the original artist.

What To Do Instead

None of this is an argument against using AI. It’s an argument for using it with intention.

A few things that actually make a difference:

Have a point of view before you open the chat. The founders who use AI well come to it with a thesis, a constraint, a specific problem. They’re not asking “what should our product strategy be”. They’re asking “here’s our strategy, what are the weakest assumptions.” AI as a sparring partner is useful. AI as a replacement for the sparring is not.

Make the weird idea survive long enough to be evaluated. The homogenization problem starts early during brainstorms, in early product discussions, in how teams communicate internally. If every rough idea gets immediately polished into something presentable, you lose the signal before you’ve had a chance to assess it. Protect the ugly first draft. The strange half-formed thought. That’s often where differentiation lives.

Read what you send. This sounds obvious. It isn’t. If you’re sending investor updates, emails, product specs, or team communications generated by AI without genuinely engaging with them first you’re not saving time, you’re spending trust. The long email that doesn’t answer the question costs more than the short one that does.

Measure outcomes, not output. Commits, content volume, email count, these are AI’s natural terrain. They go up. What matters is whether customers are staying, whether the product is getting sharper, whether the team is making better decisions than last quarter. If velocity is rising and those metrics aren’t, you have false velocity. That’s a problem worth stopping for.

Build something AI can’t instantly replicate. Proprietary data. Deep domain knowledge. Relationships. Workflow integrations that took eighteen months to get right. These are the things that survive commoditization. If your competitive advantage can be reproduced with a new system prompt, it’s not a competitive advantage.

The Startups That Survive Will Be the Ones That Stayed Human

The genericity trap isn’t malicious. Nobody woke up and decided to build something forgettable. It happens gradually, under pressure, in the small daily choices about what to think through yourself and what to hand off.

But the market doesn’t grade on intention. Investors don’t fund effort. Customers don’t stay for polish.

What they respond to, what they’ve always responded to, is something that feels like it could only have come from a specific group of people, with a specific obsession, solving a specific problem they understood better than anyone else.

AI can help you get there faster. It can’t want it for you.

The startups that figure out that distinction won’t just survive the genericity trap. They’ll be the ones that made something worth remembering, in a market full of things that looked exactly the same.

But also this can be just my complain and we will see the opposite, nothing is sure with AI.

Generative AI can now write production-ready code, produce photorealistic images, draft legal briefs, and synthesize research in seconds. What it cannot do is decide whether any of that output is actually correct, appropriate, or strategically sound. That judgment still belongs entirely to the person sitting at the keyboard, and the gap between those who can exercise it and those who cannot is becoming the defining fault line in the modern labour market.

What the data shows

The World Economic Forum’s Future of Jobs Report 2025 identifies analytical thinking as the single most sought-after core skill among employers, with seven out of ten companies considering it essential in 2025. This is followed by resilience, flexibility, and agility , so all qualities that presuppose the capacity to evaluate a situation rather than simply execute a task. Employers consistently rank critical thinking among the hardest skills to find, harder than technical skills, which AI is rapidly commoditising.

McKinsey’s 2023 research on generative AI’s economic potential estimated that current generative AI and other technologies have the potential to automate work activities absorbing 60 to 70 percent of employees’ time today — up from a previous estimate of roughly half, largely because generative AI can now handle natural language, which accounts for 25 percent of total work time. The report notes this impact falls disproportionately on knowledge work tied to higher wages and educational requirements. What remains least automatable are tasks requiring judgment, contextual reasoning, and the ability to structure ambiguous problems.

A recent IBM Institute for Business Value survey found that 4 in 5 executives believe generative AI will change employee roles and skills. Notably, the report flags that while workers at all levels will feel the effects, lower-level employees are expected to see the biggest shift which is a finding that aligns directly with the skills gap this article describes.

The real bottleneck

The problem is not that people lack access to powerful AI tools. The problem is that a tool which can produce a plausible-sounding wrong answer at high speed requires more scrutiny from its user than a slower, more transparent process ever did. Using AI responsibly is a cognitively demanding act. It requires the user to formulate a precise problem before issuing a prompt, evaluate whether the output is internally consistent and factually grounded, identify what has been omitted or hallucinated, and decide how the result fits, or fails to fit, a broader objective.

Each of these steps is an exercise in structured thinking. Without it, the user is not augmented by AI, they are simply propagating AI errors at scale.

There is a subtler risk that compounds this: the tendency to let AI reframe the problem itself. A tool that suggests an alternative approach to a task can easily nudge a user into changing business logic that was never meant to be changed. Avoiding this requires the user to enter every interaction with a clear grasp of the underlying business need, not just the immediate task, and to continually test whether the AI’s output actually serves that need or merely appears to. The prompt is not the starting point. The business requirement is.

Why the deficit is surfacing now

For decades, the ability to retrieve and process information was itself a valued professional skill. Search engines, spreadsheets, and databases rewarded those who could organise and access knowledge efficiently. AI has compressed that advantage almost to zero. What it exposes, in turn, is the underlying reasoning ability that information-handling used to mask.

Graduates who learned to research by assembling sources, without necessarily evaluating them, now face tools that assemble sources automatically. The bottleneck has moved upstream, to the question of whether the assembled result is any good. That is a question that requires logic, domain knowledge, and the willingness to be wrong , none of which AI supplies.

The professional implication

Organisations that deploy AI without investing in the critical reasoning capacity of their people are not becoming more productive. They are becoming more confident in their errors. The competitive advantage of the next decade will not accrue to those who use AI most aggressively, but to those who judge its outputs most accurately.

The labour market is beginning to price this accordingly and the split will not be subtle. Professionals who were already strong analytical thinkers are gaining a substantial productivity multiplier: AI handles the grunt work while their judgment becomes rarer and more valuable. Those who were already struggling with structured thinking face a harder reality. The tasks that once kept them employed: formatting, aggregating, drafting, summarizing , are now automated. What remains is exactly the work they found most difficult. AI did not create this gap; it made it impossible to ignore.

What to develop now

The skills worth investing in are not new but they have become urgent. First, practice decomposing problems before reaching for a tool: write down what you are actually trying to solve, what constraints apply, and what a good outcome looks like. This habit alone separates effective AI users from passive ones. Second, train yourself to challenge outputs rather than accept them: ask whether the logic holds, whether anything is missing, and whether the answer serves the original business need or has quietly drifted from it. Third, strengthen your domain knowledge, because critical evaluation without subject matter grounding is shallow. AI can surface information; only you can judge whether it is relevant and correct in context. Finally, practise structuring your thinking in writing: clear prompts, clear briefs, clear decisions. If you cannot articulate a problem precisely, you cannot verify whether it has been solved. These are not technical skills. They are the cognitive habits that make every other skill, including the use of powerful AI, actually work.

Sources

1. World Economic Forum. Future of Jobs Report 2025. Geneva: WEF, 2025. weforum.org/publications/the-future-of-jobs-report-2025/
2. McKinsey Global Institute. The Economic Potential of Generative AI: The Next Productivity Frontier. McKinsey & Company, June 2023. mckinsey.com/capabilities/mckinsey-digital/our-insights/the-economic-potential-of-generative-ai-the-next-productivity-frontier
3. IBM Institute for Business Value. Augmented Work for an Automated, AI-driven World. IBM, 2023. ibm.com/thought-leadership/institute-business-value/en-us/report/augmented-workforce