Semantic Router: The AI Traffic Controller
You have built the chatbot. You have tuned the prompt, chosen the frontier model, and wired everything into production. The responses are fluent, helpful, and impressively human. Then you look at the bill.
A third of your API calls are asking for business hours. Another chunk is saying hello, goodbye, or requesting a password reset. These are not open-ended reasoning tasks. They are simple, predictable intents with well-known answers. Yet your most expensive model is generating them token by token, one by one, thousands of times per day.
The problem is not your model. It is your routing.
Every query is treated as a full LLM problem. There is no cheap layer deciding whether a message even needs a large model. The result is predictable: latency that frustrates users, costs that scale linearly with traffic, and a system that burns money on the digital equivalent of saying “Hi there.”
A semantic router is the layer that fixes this. It is one of the simplest yet highest-return upgrades you can add to an LLM-powered product.
What Is a Semantic Router?
A semantic router is a lightweight intent classification layer that sits between the user and your LLM infrastructure. It uses a fast, cheap embedding model to compare the meaning of a query against known route definitions, then sends the query to the cheapest destination that can handle it.
The word “semantic” matters. Traditional routing relies on keywords, regular expressions, or exact matches. That works until someone asks “What time do you open?” instead of “What are your opening hours?” or “When does the store close?” The sentences share no keywords, but they share the same intent.
A semantic router captures meaning. It embeds the incoming query and the example utterances for each route into the same vector space. Then it measures cosine similarity. If the query is close enough to a known route, it goes there. If not, it falls back to the general LLM.
This makes the router both flexible and cheap. It does not need an LLM call to make a routing decision. A small sentence transformer on a CPU can classify thousands of queries per second for essentially zero cost.
Semantic Router vs. the Alternatives
Most teams start with one of two approaches. They either use keyword rules or they call an LLM to classify intent. Both have problems.
Keyword rules are brittle. The moment a user rephrases a greeting, the rule breaks. You end up with an ever-growing list of regexes that nobody wants to maintain.
LLM-as-classifier is more robust, but it defeats the purpose. You are still paying for a model call before you decide whether to call a model. The classification itself adds one to three seconds of latency and another round of token cost.
The semantic router sits in the middle. It is flexible enough to handle rephrasing, because it understands meaning. It is cheap enough to run on every query, because it uses an embedding model rather than a generative model. And it is fast enough that the routing decision itself is measured in milliseconds.
The Five Steps of Semantic Routing
The core idea is simple, but the implementation has a clear structure. Here is the flow from route definition to final destination.
1. Define Routes
A route is a destination with a set of example utterances. Static routes return hardcoded answers. Agent routes forward to a specialized model or system. You usually want a fallback route for anything uncertain.
static routes:
greeting: ["hello", "hi", "hey there"]
hours: ["what time do you open?", "when do you close?"]
farewell: ["bye", "goodbye", "see you"]
agent routes:
billing: ["i was charged twice", "refund my subscription"]
technical: ["my integration is failing", "api error 500"]
fallback: general LLM2. Embed Route Utterances
At startup, encode every example utterance for every route using a small sentence transformer. The result is a matrix of normalized embedding vectors. This is done once, so the cost is amortized across every query.
3. Embed the Incoming Query
When a user sends a message, encode that single query with the same model. The embedding should be normalized so cosine similarity is just a dot product.
4. Score with Cosine Similarity
Compare the query vector against every route vector. The highest score is the winning route. You can use the maximum score across utterances, the average score, or a weighted blend. Maximum is the simplest starting point and tends to be most decisive.
5. Route by Threshold
If the best score is above your threshold, send the query to that route. If not, send it to the fallback. The threshold is the most important hyperparameter in the system. Too low and queries go to the wrong route. Too high and the LLM is doing work a static route could have handled. This one is kinda tricky to be honest.
The Cost of Saying Hello
Let me make the economics concrete. Imagine a customer support bot that handles one hundred thousand messages per month.
Roughly thirty-five percent of those messages are greetings, farewells, business hours, password resets, and other simple FAQs. You almost definitely use the top tier model for this Customer Support Agent since of course you don’t want less intelligence model ruin your potential customer experience. At a frontier model price of fifteen dollars per million tokens and an average of one hundred fifty tokens per query, that trivial traffic alone costs around fifteen dollars per day.
That is over five thousand dollars per year spent generating answers that were already known.
And cost is only half the story. Each LLM call adds one to three seconds of latency. A user asking for your opening hours has to wait for a model to slowly type out “We are open Monday to Friday, 9 AM to 5 PM.” That is a frustrating experience for a question that deserves an instant answer.
A semantic router changes the equation. The embedding model is local and runs in under ten milliseconds. Static routes return in microseconds. The expensive model only sees the queries that actually need it.
With the router handling thirty-five percent of traffic as static routes, daily LLM spend drops from roughly forty-five dollars to twenty-nine dollars. Responses return in under ten milliseconds instead of one to three seconds. Over a year, the savings on a single bot approach six thousand dollars. And the user experience on simple queries improves dramatically.
Two Kinds of Destinations
Routes are not all the same. The most useful systems distinguish between static routes and agent routes.
- Static routes return pre-written responses. They are perfect for greetings, business hours, refund policies, and anything with a deterministic answer. They cost zero tokens and zero milliseconds. The router does not call a model at all.
- Agent routes forward the query to a specialized model or pipeline. The sales agent might have a prompt tuned for persuasive, helpful responses. The billing agent might have access to account data. The technical support agent might be allowed to run diagnostics. Each route gets its own context, tools, and system prompt.
The key insight is that the router decides which destination fits without calling an LLM to make that decision. It is classification, not generation. That distinction is what makes the cost and latency numbers work.
You can also add a safety route before the main router. This route catches off-topic, abusive, or policy-violating inputs before they reach any LLM. Because it is cheap, you can run it on every message without worrying about the bill.
Build a Semantic Router in Python
Let’s make it into a simple code implementation. You need a sentence transformer, some route definitions, cosine similarity, and a threshold. Here is a complete example you can adapt.
import numpy as np
from sentence_transformers import SentenceTransformer
class SemanticRouter:
def __init__(self, routes, model_name="all-MiniLM-L6-v2", threshold=0.8):
self.routes = routes
self.threshold = threshold
self.encoder = SentenceTransformer(model_name)
# Pre-compute normalized embeddings for every route utterance.
# This is done once at startup so each query only needs one embedding.
for route in self.routes:
route["emb"] = self.encoder.encode(
route["utterances"],
normalize_embeddings=True,
)
def route(self, query):
# Encode the incoming query with the same model.
q_emb = self.encoder.encode(query, normalize_embeddings=True)
# Compare against each route using cosine similarity.
# Because vectors are normalized, cosine similarity is just a dot product.
# We take the maximum score across all utterances in a route.
scores = [
(route["name"], np.max(route["emb"] @ q_emb))
for route in self.routes
]
name, score = max(scores, key=lambda x: x[1])
return name if score >= self.threshold else "fallback"
if __name__ == "__main__":
routes = [
{"name": "greeting", "utterances": ["hello", "hi", "hey", "hi there"]},
{"name": "farewell", "utterances": ["bye", "goodbye", "see you", "it's time to go"]},
{
"name": "hours",
"utterances": [
"what are your hours",
"when do you open",
"when does the store close",
],
},
]
router = SemanticRouter(routes, threshold=0.75)
queries = [
"Hi there!",
"It's time to go.",
"What time do you open?",
"Who won the game?",
]
for q in queries:
print(f"Query: '{q}' -> Routed to: {router.route(q)}")The code is intentionally simple. There is no vector database, no LangChain, no external API. The model is twenty-two megabytes, runs on CPU, and the only dependency beyond NumPy is the sentence transformer library.
Get Cikal Merdeka’s stories in your inbox
Join Medium for free to get updates from this writer.
When you run this, the output looks like this:
Query: 'Hi there!' -> Routed to: greeting
Query: 'It's time to go.' -> Routed to: farewell
Query: 'What time do you open?' -> Routed to: hours
Query: 'Who won the game?' -> Routed to: fallbackNotice that “What time do you open?” routes to the hours intent even though it does not share a single keyword with the example “When does the store close?” That is the semantic part in action.
Production Patterns: Layered Routing and Guardrails
For larger systems, one router is not enough. The most robust architectures use stacked routers and guardrails.
- Layered routing works like a decision tree. A top-level domain router first decides whether the query is about billing, sales, or technical support. Then a sub-router within that domain decides whether the query is a static FAQ or a problem that needs an agent. This keeps each router small and focused, which improves accuracy and makes maintenance easier.
- Safety routing is a separate router that runs first. It catches inputs that should never reach an LLM: off-topic questions, policy violations, or jailbreak attempts. Because it is cheap, you can be aggressive about what you block. A false positive here just sends a user to a human fallback. A false negative might expose your model to abuse.
- Semantic caching pairs naturally with routing. If the router classifies a query to a static route, and a similar query has been answered before, you can return the stored answer without any model call at all. This is especially powerful for FAQs that are semantically similar but not identical.
Choosing the Right Embedding Model and Threshold
The most common mistake is overthinking the model. For most production routing tasks, all-MiniLM-L6-v2 is the right choice. It is twenty-two megabytes, runs on CPU, costs nothing, and classifies intent well enough to be useful. It has only 384 dimensions, so similarity computation is fast.
Only upgrade to a larger or API-based embedding model if you can measure a meaningful accuracy gap on your labeled evaluation set. The extra dimensions and quality usually do not matter for routing, and they add latency and cost.
The threshold is more important than the model. It controls the trade-off between routing precision and LLM call rate.
Tune the threshold on a labeled set of real queries. Plot the precision and the LLM call rate as you sweep the threshold. The goal is to maximize the percentage of trivial queries handled by static routes while keeping the misroute rate below your tolerance.
You should also measure the router in production. Log the query, the winning route, the score, and the final outcome. Every week, review low-confidence queries and ambiguous cases. Add new example utterances to the routes that are underperforming. A semantic router is not a set-and-forget component; it improves with feedback.
Real-World Workflow Example: Customer Support Bot
Let me put this together in a concrete scenario. Imagine a SaaS company with a support chatbot that currently routes everything to a frontier model. A semantic router is deployed with five routes:
The static routes save the company forty-five percent of LLM calls immediately. The agent routes improve quality because each one gets a tailored system prompt and the right tools. The general LLM only handles the ten percent of queries that actually need it.
Latency drops because static responses return in under ten milliseconds. The billing and technical agents are faster because they receive shorter, more focused prompts rather than a long general system prompt.
The implementation cost is small. The model is open source. The code is less than a hundred lines. The deployment can be a single Python service or a lightweight function inside your existing API.
Guardrails Are Just Another Route
The customer support bot example has one more route worth adding: a safety route. Some inputs should never reach any LLM, no matter how cheap the routing step is. A semantic router can block off-topic, abusive, or policy-violating queries before they reach the billing or technical agents.
The guardrail route works exactly like the others. You define example utterances for the kinds of inputs you want to reject: prompts that ask for medical advice, legal opinions, competitors’ proprietary data, or anything outside your product’s scope. When the router matches the query to this route, it returns a standard refusal instead of forwarding the request.
The advantage is speed and cost. A safety check that runs inside a small embedding model is fast enough to run on every message and cheap enough that you never think about the bill. A false positive just sends a user to a human fallback. A false negative sends a risky query to your model.
In practice, the safety router runs first. The query is embedded, scored against the safety route, and only released to the domain router if it passes. This is the same pattern as layered routing: classify cheaply, decide early, and keep the expensive model away from work it should never do.
When to Use a Semantic Router
A semantic router is not a universal solution. It shines when you have predictable, high-volume intents mixed with genuinely open-ended queries. It is less useful when every query is unique and requires reasoning.
Use it when:
- A meaningful share of your traffic is greetings, FAQs, or other simple intents.
- You have multiple agents or pipelines that each need a different prompt or tools.
- You want to reduce LLM cost without degrading user experience.
- You need a fast, local safety check before sending queries to a model.
Skip it when:
- Every user message is a unique, open-ended question.
- Your intent categories are too vague to define with example utterances.
- You have no labeled data or way to evaluate routing accuracy.
Final Thoughts
The best LLM architecture is not the one that uses the biggest model for every query. It is the one that knows exactly when not to use it.
A semantic router turns your LLM pipeline from a flat, expensive firehose into a targeted, efficient system. It intercepts trivial queries before they reach the model. It sends complex queries to the right specialized agent. It leaves the frontier model for the work that genuinely needs it.
This is the pattern that makes LLM products economically viable at scale. It saves money. It improves latency. It makes the user experience better. And it does all of this with a small, open-source embedding model and a few lines of Python.
Here are several key takeaways from this article:
- A semantic router is an intent classifier, not a generator. It uses cheap embeddings to decide where a query should go before any expensive LLM is called.
- Route by meaning, not keywords. Embeddings capture rephrasing and synonyms that keyword rules miss.
- Static routes are your highest-ROI optimization. Returning pre-written answers for greetings, FAQs, and farewells eliminates both token cost and model latency.
- Agent routes improve quality. Specialized models with focused prompts and the right tools often outperform a single general model.
- Threshold tuning is the critical hyperparameter. Sweep scores on a labeled evaluation set to balance routing precision against LLM call rate.
- Layer routers for larger systems. Use a safety router first, then domain routers, then sub-routers for focused intent classification.
- Start with
all-MiniLM-L6-v2. It is small, free, fast, and accurate enough for most routing tasks. Only upgrade if you measure a real gap.
Thank you for reading this article! I hope you found it helpful.
#AIEngineering #LLM #SemanticRouter #MachineLearning #MLOps #GenerativeAI #CostOptimization #NaturalLanguageProcessing
