# How do we cut our AI running costs?

_Running AI in production · Last updated 2026-08-20 · Bitfumes_

## Short answer

Measure cost per transaction before changing anything, because a monthly total tells you nothing about which feature is expensive. In most production systems the majority of spend is input tokens — retrieved context and long system prompts — rather than the model's output, so the highest-return fixes are trimming what you send, caching the stable prefix of every prompt, and routing easy requests to a smaller model. Those three commonly halve a bill without touching answer quality, and all of them are cheaper than renegotiating rates. Do not optimise a feature whose behaviour is still changing weekly: distillation and model downgrades cost more engineering time than they save until the scope has settled.

## First, find out what you are paying for

Nearly every team that thinks it has a cost problem actually has a measurement problem. A provider invoice is one number for the whole company; the question you need answered is which feature, which call, and which half of the call.

- Log input tokens and output tokens separately on every call. They are priced differently and the split tells you which lever applies.
- Tag every call with the feature that made it, so spend can be grouped the way the business thinks about it.
- Report cost per transaction — per ticket, per document, per quote — not cost per month. Only the per-transaction figure survives a change in volume.
- Track the p95 as well as the mean. A small tail of enormous prompts is a common and invisible cause of a bill that will not come down.

## The levers, in order of return

| Lever | Typical effect | Cost to implement |
| --- | --- | --- |
| Trim retrieved context — fewer, better chunks plus re-ranking | Large | Low; usually improves quality at the same time |
| Prompt caching on a stable prefix | Large where prefixes repeat | Low, but requires arranging the prompt so the stable part comes first |
| Route easy requests to a smaller model | Large | Medium — needs an evaluation set to prove the cheap path is safe |
| Cap and structure outputs | Medium | Low |
| Batch or defer non-interactive work | Medium | Low, where the work does not need an immediate answer |
| Distil a small model for a fixed task | Large | High; only worth it once the behaviour has stopped changing |
| Self-host an open-weight model | Varies | High — trades a per-token bill for GPU capacity and operations |
| Committed-use or volume pricing | Medium | Low, but only available once your volume is real |

The ordering is deliberate. The first two cut cost without any risk to quality, which means they need no evaluation work to justify; everything below the third line changes what the model sees or is, and should not be attempted without a way to detect that answers got worse.

## The habits that make bills grow

- Sending a whole document when three paragraphs would answer the question.
- Retrieving the top twenty chunks 'just in case' — which pays twenty times over and often lowers answer quality by burying the relevant passage.
- Unbounded conversation history, so a long session costs quadratically in the length of the session.
- Retries without backoff or a cap, which turn a provider blip into a bill.
- Agent loops with no step limit, where a failing tool call is retried until something intervenes.
- No per-feature budget or alert, so the first signal is the invoice.

## When cheaper is the wrong answer

Inference is not always the expensive part. If a support assistant costs eight cents per ticket and saves six minutes of an agent's time, the correct response to a rising bill is to send it more tickets. Compare inference spend against the labour or revenue it moves before treating it as a cost to minimise — and if that comparison cannot be made because nobody recorded a baseline, that is the more urgent problem.

## Frequently asked

### What share of a typical LLM bill is input versus output?

It varies by workload, but retrieval-augmented systems are usually dominated by input tokens, because every call carries retrieved passages and a system prompt while returning a few hundred tokens of answer. Instrument your own split before acting — it decides which lever applies.

### Does prompt caching work for every workload?

It helps wherever a substantial prefix repeats across calls — a long system prompt, a fixed policy document, a stable tool schema. It does nothing for calls that are unique end to end, and it requires the stable content to sit at the front of the prompt, which is a code change rather than a setting.

### Is self-hosting cheaper than an API?

Only at sustained high volume, and only once you count GPU capacity, idle time, and the engineers who keep it running. The sound reasons to self-host are data residency, regulation and vendor independence; treat any cost saving as a bonus rather than the business case.

### How much can we expect to save?

We will not quote you a percentage for a system we have not measured. What we can say is that the first three levers in the table above are the ones that most often produce a large reduction, and that they are usually available because nobody instrumented the system finely enough to see them.

---

Source: https://bitfumes.com/answers/how-to-reduce-ai-costs
Written by Bitfumes, an AI consultancy founded in 2017 by Sarthak Shrivastava (Docker Captain). Entry point: https://bitfumes.com/ai-assessment