How does smart routing between multiple LLMs work in a chatbot?
In an LLM routing chatbot, a decision layer analyzes each message and sends it to the most suitable model, so simple questions go to a fast, cheap model and complex ones to a stronger model.
In an LLM routing chatbot, a decision layer sits between your bot and the models: before answering, it analyzes each message (type, difficulty, length and language) and sends it to the most suitable model using rules, a classifier or similarity to known examples. Simple questions go to a fast, cheap model; complex ones to a stronger model.
If a model fails, the request moves to another one. That way the chatbot answers better, faster and for less than if it used a single model for everything.
What problem does routing solve in a chatbot?
A customer service chatbot receives very different messages: “hi”, “what are your hours?”, “compare these two plans for my situation” or “summarize this 20-page contract”. If everything goes through the most powerful model, you overpay and answer more slowly on the easy messages. If everything goes through the cheapest model, you fail on the hard ones.
Routing solves that trade-off between quality, cost and latency by deciding, message by message, which model answers. It isn't the same as classic load balancing: a load balancer spreads traffic across identical copies of a server, while an AI router chooses between different models that answer differently, cost differently and take different amounts of time. If you want to see which services exist for this, check our comparison of services for routing queries between language models. Here we focus on how it works under the hood.
The parts of a routing system
- A single entry point. Your chatbot sends every request to one API instead of integrating each provider separately. That entry point usually follows the OpenAI API format, which is the de facto standard today.
- A query analyzer. It extracts signals from the message: intent, length, whether it includes code or a document, language, conversation history and whether it needs data about your business.
- A decision policy. With those signals it picks the model. It can be a fixed rule, a classifier or a trained router (more on these below).
- A model catalog. The list of available models with what you know about each one: which tasks it handles well, how long it takes and how much it costs.
- Fallback and retries. If a model returns an error, takes too long or the provider is down, the request moves to another model.
- Logging and measurement. It records which model answered, how much it cost and how long it took, so you can tune the rules with real data.
The most common routing techniques
1. Rule-based routing
This is the starting point. You write simple conditions: if the message is under a certain length and is a greeting or a frequently asked question, it goes to the fast model; if it includes an attachment or asks for an analysis, it goes to the large model; if it asks for code, it goes to a model specialized in programming. It's easy to understand and debug, but it falls short when messages are ambiguous.
2. Classifier-based routing
A small, cheap model reads the message and labels it (for example “greeting”, “pricing question”, “technical support”, “complex reasoning”). Based on the label, the system picks the model that answers. The classifier adds an extra call, so it should be very fast; in exchange, it handles the variety of natural language much better than fixed rules.
3. Semantic routing with embeddings
You turn the message into a numeric vector (an embedding) and compare it with vectors of examples you have already classified. If the message closely resembles your “billing questions” examples, it goes to the billing route. It's fast, doesn't need a call to a large model and is the idea behind open-source libraries such as Semantic Router.
4. Quality-trained routers
A model learns, from comparisons between answers, when a cheap model is enough and when a strong one is needed. The open-source project RouteLLM and services such as Not Diamond or Martian follow this approach. It works well at high volume, but you need to measure carefully whether it really maintains quality in your case.
5. Cascade
The cheap model answers first. If the answer fails a check (for example, the model says it isn't sure, the answer doesn't match the expected format or a verifier rejects it), the question is sent again to a more powerful model. It saves money on easy cases, but on hard ones it adds the latency of two calls.
6. Fallback
It doesn't choose by quality but by availability: you define an order of models and, if the first one fails or exceeds a timeout, the request goes to the next. It should be part of any chatbot serving real customers, combined with any of the techniques above.
What a message's flow looks like, step by step
- The user types into the chatbot on your site or app.
- Your server (never the browser) receives the message and builds the context: recent history and, if relevant, information about your business.
- The routing layer classifies the message with rules, a classifier or embeddings.
- It picks the model and sends the request through the unified API, changing only the model name.
- If there's an error or a delay, it applies the fallback with the next model on the list.
- It returns the answer to the user and logs the model, cost and time.
- Every so often you review those logs and adjust the rules: which queries could go to a cheaper model and which need a better one.
How to implement it without overcomplicating things
- Start with two models and one rule. A fast model for everyday questions and a powerful one for complex ones. Measure before adding more routes.
- Use a single API for all models. With a service like Dongee AI Router, a unified API for several AI models, you get multiple models behind a single key, with priority on fast models hosted in Dongee's data center in Canada. If your code already uses the OpenAI API or a compatible library, you only change the base URL and the key; routing to another model means changing one parameter.
- Keep keys on the server and create one per project, limited to the models that project needs. A key in the browser can be seen by anyone and used to spend your balance.
- Add fallback from day one with a maximum timeout and a second model.
- Control spending. With a prepaid model like Dongee AI Router's (from 3 USD, no monthly fee; new accounts get 3 USD in free starter credit), requests stop when your balance runs out instead of producing a surprise bill. Review usage by model and by key.
- Host the logic close to your users. The routing layer is your own code running on a server. For a simple chatbot, Business Hosting is enough; if you need more control or your own processes, use a Dongee Cloud server with root access. If the chatbot lives inside an app you built with AI, you can publish it with Dongee Live, Dongee's hosting for publishing Node.js apps and static sites.
An honest comparison of approaches
| Approach | When it makes sense | Advantages | Things to keep in mind |
|---|---|---|---|
| Your own rules + a unified API (for example, Dongee AI Router) | Business chatbots getting started, with few query types | Simple, predictable, easy to debug; one key and prepaid billing | You write and maintain the rules; with highly varied messages they fall short |
| Your own classifier or embeddings | Teams with some technical experience and varied queries | Handles natural language better; semantic routing is fast | You need labeled examples and must review classification errors |
| Self-hosted open-source gateway (for example, LiteLLM) | Technical teams that want full control over fallbacks and budgets | Configurable, no extra intermediary | You maintain the server, updates and security; each provider is paid separately |
| Trained routers (RouteLLM, Not Diamond, Martian) | High volume, where every point of cost matters | They pick the model per query automatically | Another layer to measure; you must validate that quality doesn't drop in your case |
| A single model for everything | Prototypes or very little traffic | The simplest option | You overpay on easy queries or fail on hard ones, and without fallback the chatbot goes down when the provider does |
In short: for most businesses, the sensible path is to start with simple rules on top of a unified API, add fallback and measurement, and only move to classifiers or trained routers later if the volume justifies it.
Common mistakes
- Routing without measuring. Without logs of cost, time and quality per model, you don't know whether your routing helps or hurts.
- A classifier slower than the savings. If deciding takes almost as long as answering, you lose the speed advantage.
- Switching models mid-conversation without handling context. Send the history needed so the second model doesn't lose the thread.
- Forgetting fallback. Every provider has outages; a customer service chatbot shouldn't depend on just one.
- Exposing keys in the frontend. Every call to a model must come from your server.
Frequently asked questions
Does smart routing improve quality or only lower costs?
Both, if it's configured well. It improves quality because it sends hard tasks to the model that handles them best, and it lowers costs and response times because a lighter model handles the easy ones.
Do I need to know how to train models to route?
No. You can start with simple rules and a unified API. Training only comes in if you later use a quality-trained router.
What's the difference between an AI router and a load balancer?
A load balancer spreads traffic across identical servers. An AI router chooses between different models based on how well, how fast and how cheaply each one handles the query.
Can I use Dongee AI Router with my existing code?
If your code uses the OpenAI API or a compatible library, yes: you change the base URL and the key. You need to be a Dongee customer and activate it from your client area.
Does Dongee store my chatbot's conversations?
According to the Dongee AI Router page, only the billing data of each request is logged, never its content.
How much does routing with Dongee AI Router cost?
It's prepaid from 3 USD, with no monthly fee, and new accounts get 3 USD in free starter credit: you're only charged for what you use. If your balance runs out, requests stop until you top up, with no debt and no automatic charges without your authorization. See the details on the Dongee AI Router page.
Does it work for chatbots in any country?
Yes. The technique is the same anywhere; what changes is where you host your server and which currency you pay in. Dongee AI Router can be purchased from the United States or Brazil with no country restrictions, and you can pay in USD, EUR, BRL, COP, MXN, PEN or CLP. For more infrastructure options, see our guide to cloud servers with AI tools.
Comenta lo que quieras
Unete a la charla
Solo ingresa tu email