Every few months a new model tops the leaderboards, and every few months teams migrate for the wrong reasons. A model that's 2% better on a general reasoning benchmark can still be the worse choice for your product once you weigh latency, cost per call, and how consistent it is on your specific task.
What we actually evaluate on client work
- Task accuracy on a real, held-out sample of your own data — not a public benchmark.
- Latency at the p95, not the average, because that's what your slowest user experiences.
- Cost per successful outcome, factoring in retries and fallback calls, not just cost per token.
- Consistency across runs — how often does the same input produce a meaningfully different output.
- How gracefully it fails, and whether a smaller fallback model can catch the failure cheaply.
In practice this usually means running two or three candidate models against a small evaluation set built from your own data before writing a line of production code — and often means using different models for different steps of a single pipeline, rather than one model for everything.
The right model is the one that's cheapest and fastest at the accuracy bar your product actually needs — not the one on top of the leaderboard.