In an AI application, a lot of calls to a language model are doing something very small: deciding which of about seven things the user wants, so the right code path runs. That decision has a fixed set of answers, it does not require world knowledge, and it does not need to generate anything.
Sending it to a general-purpose model works. It also costs a network round trip, a per-token charge, and the property that the same input reliably produces the same output. A fine-tuned classifier a few hundred times smaller does that job better on every one of those axes.
This is design and research work rather than a shipped result. I have no production accuracy numbers, and where I mention metrics they are the ones I would measure, not ones I am reporting.
What the prompt is costing
Four things, and only the first is obvious.
Latency, in the tens to hundreds of milliseconds, on a decision that has to happen before any real work starts. It sits at the front of every request.
Cost that scales with traffic, on a call that produces one word of output.
Non-determinism. The same phrasing can classify differently across calls, and across model versions you did not choose to upgrade to. Debugging a routing bug that reproduces four times out of five is a genuinely bad afternoon.
Prompt sensitivity. Adding an eighth intent means rewriting a prompt that was tuned for seven, and the boundary between two existing intents can shift as a side effect. There is no regression test for a paragraph of English.
On constrained hardware the picture gets worse. If you are running the model locally, as I do, the round trip is not tens of milliseconds. It is seconds.
The alternative
A compact transformer classifier, fine-tuned on your intent set:
search_business
compare_businesses
get_directions
filter_by_price
filter_by_rating
ask_opening_hours
unknownSmall enough to run on CPU in single-digit milliseconds, deterministic, and it returns a probability distribution rather than a word, which turns out to be the important part.
The bit that makes it work: route on confidence
A classifier that must always answer is a classifier that will confidently mislabel anything outside its training distribution. Users say unexpected things constantly. So the classifier does not decide, it proposes, and confidence decides:
user input
│
v
intent classifier (a few ms, CPU)
│
├── high confidence ──────► deterministic handler
│
└── low confidence ───────► LLM reasoning, or ask a clarifying questionMost traffic is ordinary and takes the cheap path. Genuinely ambiguous or novel input escalates to the expensive one, which is exactly what the expensive one is good at. You are not replacing the language model. You are stopping it from being the front door.
That unknown class in the intent list is not a leftover. It is trained on
deliberate out-of-domain examples so the model can learn what is not its job,
which is what makes the confidence threshold meaningful rather than a threshold
over seven forced choices.
Where the work really is
Fine-tuning is the easy part. These are what determine whether it works:
Define the taxonomy first, and keep it small and stable. Intents that overlap semantically cannot be separated by any model, because the training labels contradict each other. If two intents route to the same handler, they are one intent.
Collect real phrasings, not phrasings you invented. The set you write at your desk is a narrow, grammatical, well-spelled subset of what users type.
Add hard negatives and ambiguous examples on purpose. The decision boundary is learned from examples near it. Fifty clear cases teach less than five genuinely borderline ones.
Split by semantic pattern, not randomly. This is the one that quietly invalidates results. If “find me a cafe” is in training and “find me a restaurant” is in test, your test set is measuring memorisation. Split so that patterns do not appear on both sides, and expect the honest number to be lower than the flattering one.
Calibrate the threshold, do not guess it. The threshold decides how much traffic escalates, which is the entire cost and quality trade-off. Set it from a confusion matrix, then revisit it with real traffic.
The metrics I would hold this to: macro F1, so a rare intent cannot be hidden by a common one. Per-intent recall, to see which one is being swallowed by its neighbour. A confusion matrix, since which pairs get confused tells you whether the taxonomy is wrong. Out-of-domain rejection rate. p50 and p95 latency. Memory footprint. And confidence calibration, because a threshold over uncalibrated probabilities is a threshold over noise.
When this is the wrong call
I would not do this for a decision that needs world knowledge, an intent set that is still changing weekly, low traffic where the prompt costs nothing at your volume, or a case where the output is genuinely open-ended rather than one of N. Training and maintaining a model is real work with a real half-life, and below some traffic level the prompt is simply cheaper in engineer-hours.
The trigger is a decision that is closed-set, high-frequency, and latency-sensitive. All three. Missing any one of them and the prompt is probably fine.
The general principle
The framing I keep coming back to:
A language model is a general-purpose tool being used for a special-purpose task. Sometimes generality is what you are paying for. On a seven-way routing decision, it is pure overhead.
The same reasoning applies well outside intent classification. Extracting a date from a string, checking whether text is English, detecting profanity, deduplicating near-identical records, deciding whether an email is a receipt. Each has a small, fast, deterministic solution that predates language models and still works.
Use the model for the parts that need judgement. Let cheaper things handle the parts that need a decision.