Running AI in production

How do we cut our AI running costs?

Short answer

Measure cost per transaction before changing anything, because a monthly total tells you nothing about which feature is expensive. In most production systems the majority of spend is input tokens — retrieved context and long system prompts — rather than the model's output, so the highest-return fixes are trimming what you send, caching the stable prefix of every prompt, and routing easy requests to a smaller model. Those three commonly halve a bill without touching answer quality, and all of them are cheaper than renegotiating rates. Do not optimise a feature whose behaviour is still changing weekly: distillation and model downgrades cost more engineering time than they save until the scope has settled.

Last updated August 20, 2026 · Bitfumes AI consultancy · view as Markdown

First, find out what you are paying for

Nearly every team that thinks it has a cost problem actually has a measurement problem. A provider invoice is one number for the whole company; the question you need answered is which feature, which call, and which half of the call.

Log input tokens and output tokens separately on every call. They are priced differently and the split tells you which lever applies.
Tag every call with the feature that made it, so spend can be grouped the way the business thinks about it.
Report cost per transaction — per ticket, per document, per quote — not cost per month. Only the per-transaction figure survives a change in volume.
Track the p95 as well as the mean. A small tail of enormous prompts is a common and invisible cause of a bill that will not come down.

The levers, in order of return

Lever	Typical effect	Cost to implement
Trim retrieved context — fewer, better chunks plus re-ranking	Large	Low; usually improves quality at the same time
Prompt caching on a stable prefix	Large where prefixes repeat	Low, but requires arranging the prompt so the stable part comes first
Route easy requests to a smaller model	Large	Medium — needs an evaluation set to prove the cheap path is safe
Cap and structure outputs	Medium	Low
Batch or defer non-interactive work	Medium	Low, where the work does not need an immediate answer
Distil a small model for a fixed task	Large	High; only worth it once the behaviour has stopped changing
Self-host an open-weight model	Varies	High — trades a per-token bill for GPU capacity and operations
Committed-use or volume pricing	Medium	Low, but only available once your volume is real

The ordering is deliberate. The first two cut cost without any risk to quality, which means they need no evaluation work to justify; everything below the third line changes what the model sees or is, and should not be attempted without a way to detect that answers got worse.

The habits that make bills grow

Sending a whole document when three paragraphs would answer the question.
Retrieving the top twenty chunks 'just in case' — which pays twenty times over and often lowers answer quality by burying the relevant passage.
Unbounded conversation history, so a long session costs quadratically in the length of the session.
Retries without backoff or a cap, which turn a provider blip into a bill.
Agent loops with no step limit, where a failing tool call is retried until something intervenes.
No per-feature budget or alert, so the first signal is the invoice.

When cheaper is the wrong answer

Inference is not always the expensive part. If a support assistant costs eight cents per ticket and saves six minutes of an agent's time, the correct response to a rising bill is to send it more tickets. Compare inference spend against the labour or revenue it moves before treating it as a cost to minimise — and if that comparison cannot be made because nobody recorded a baseline, that is the more urgent problem.

Frequently asked

What share of a typical LLM bill is input versus output?

It varies by workload, but retrieval-augmented systems are usually dominated by input tokens, because every call carries retrieved passages and a system prompt while returning a few hundred tokens of answer. Instrument your own split before acting — it decides which lever applies.

Does prompt caching work for every workload?

It helps wherever a substantial prefix repeats across calls — a long system prompt, a fixed policy document, a stable tool schema. It does nothing for calls that are unique end to end, and it requires the stable content to sit at the front of the prompt, which is a code change rather than a setting.

Is self-hosting cheaper than an API?

Only at sustained high volume, and only once you count GPU capacity, idle time, and the engineers who keep it running. The sound reasons to self-host are data residency, regulation and vendor independence; treat any cost saving as a bonus rather than the business case.

How much can we expect to save?

We will not quote you a percentage for a system we have not measured. What we can say is that the first three levers in the table above are the ones that most often produce a large reduction, and that they are usually available because nobody instrumented the system finely enough to see them.