Running AI in production

Which AI model should we use?

Short answer

Choose by task and constraint rather than by leaderboard position. Prove the use case works on a frontier model first, then move down the price ladder to the cheapest model that still passes your own evaluation set — for classification, routing and extraction that is usually a small model, and for long multi-step reasoning it usually is not. The decision that actually matters is not which vendor you pick but whether you built an evaluation set before picking, because without one every comparison is a matter of taste. Keep the model behind an abstraction so switching is a configuration change, since the ranking will move again within months.

Last updated August 20, 2026 · Bitfumes AI consultancy · view as Markdown

Start with the constraint, not the model

Model choice is over-discussed because it feels like the decision. In practice the constraints below eliminate most of the field before capability is considered at all, and the remaining choice is narrow enough to settle with a day of measurement.

Accuracy bar — what happens when the answer is wrong, and whether a human sees it before it takes effect.
Latency budget — agreed as a percentile, not an average. A p95 of four seconds means one call in twenty is slower than that.
Cost per transaction at real volume, not cost per million tokens on a pricing page.
Data boundary — whether the data may leave your infrastructure at all, which is the one constraint that can force an open-weight model regardless of capability.
Modality — whether the input is clean text, or scans, screenshots, photographs and audio.
Tool-use reliability, if the feature calls functions. This varies more between models than raw reasoning does, and it is what breaks agents.

The model tier that usually wins, by task

Task	Tier that usually wins	Why
Classification, routing, extraction from clean text	Small / cheap tier	The task is narrow — a larger model buys nothing your evaluation set can detect
Messy PDFs, scans, screenshots, forms	Mid-tier multimodal	Vision is the requirement; frontier reasoning usually is not
Long multi-step reasoning, code generation, agents	Frontier tier	Errors compound across steps, so per-step quality dominates total cost
High volume, fixed scope, latency-sensitive	Small tier, or a distilled model	Once behaviour has stopped changing, cost and p95 latency dominate
Regulated data that cannot leave your infrastructure	Open-weight, self-hosted	Control is the requirement; the capability gap is the price of it

Note what is missing from that table: brand names. Any specific recommendation would be stale before you finished reading it, and the tiers are stable in a way the leaderboard is not.

Benchmarks are a filter, not a decision

Public benchmarks are useful for excluding models that are obviously unsuitable and useless for choosing between the top few. They measure general capability on tasks that are not yours, on data that may have leaked into training, at settings nobody discloses. A model two points down a leaderboard routinely wins on the only test that matters, which is a hundred of your own inputs scored against your own definition of a good answer.

Building that test is a day or two of work and it is the cheapest insurance in the project. It also converts every future model release from an anxious re-evaluation into a script you run.

Build these four things and the choice stays cheap

An abstraction layer, so the model is named in one configuration file rather than in forty call sites.
An evaluation set that runs in CI, so a model swap is a pull request with a number attached.
Model version recorded with every logged call, so a quality complaint can be traced to what actually served it.
Cost per transaction on a dashboard, broken down by feature, so a regression in spend is visible before the invoice.

Teams with these four switch models in an afternoon. Teams without them treat every provider announcement as a threat, and end up on whatever they picked first.

How Bitfumes approaches it

We build the evaluation set before we argue about models, on your data and with your definition of a correct answer, and we keep the model behind an interface from the first commit. Where a smaller model passes, we use the smaller model and say so — the alternative is billing you for frontier inference on a task that never needed it.

Frequently asked

Should we standardise on one model for everything?

One provider is a reasonable default for contracting and support, but one model rarely is. Most production systems end up with a small model doing routing and extraction and a larger one doing the part that genuinely needs reasoning, because the cost difference between them is often tenfold.

How often should we revisit the model choice?

Re-run your evaluation set whenever a provider ships a relevant model, and otherwise quarterly. The point of an automated evaluation set is that revisiting costs an afternoon rather than a project, so there is no reason to defer it.

Are open-weight models good enough now?

For classification, extraction, summarisation and most retrieval-augmented answering, yes. For long agentic chains and hard reasoning they still trail the frontier tier. The honest reason to self-host is control — residency, regulation, removing a vendor dependency — rather than cost, because you trade a per-token bill for GPU capacity and an operations burden.

Do we need a router that picks a model per request?

Not at the start. Route by difficulty only once you have an evaluation set proving the cheap path is safe for the requests you send down it, otherwise you have added a second system that can be wrong. Static routing by feature captures most of the saving with none of the complexity.