# AI glossary for people buying AI _39 terms · Bitfumes_ Plain definitions of the terms that appear in AI proposals, written for the person signing the contract rather than the person writing the code. Each entry says what the thing is, then what it changes about a buying decision. ## Core concepts ### Large language model (LLM) (also: LLM, foundation model) A large language model is a neural network trained on very large amounts of text to predict what token comes next, which turns out to be enough to summarise, translate, classify, write code and follow instructions. It has no memory between calls and no access to your systems unless you give it one. **Why it matters when buying:** Almost every disappointment in an AI project traces back to forgetting the second sentence. The model does not know your business, and closing that gap is the engineering work — the model itself is the part you buy off the shelf. ### Token A token is the unit a language model reads and writes — roughly three-quarters of an English word, so 1,000 tokens is about 750 words. Model pricing, context limits and latency are all measured in tokens rather than words or characters. **Why it matters when buying:** Every cost estimate you are given is a token estimate wearing a disguise. Ask what token volume a quoted monthly figure assumes, because the same feature can differ tenfold in cost depending on how much context each call carries. More: https://bitfumes.com/answers/how-to-reduce-ai-costs ### Context window The context window is the maximum number of tokens a model can consider in a single call, counting the system prompt, the conversation, any retrieved documents and the answer it produces. Anything outside it does not exist as far as the model is concerned. **Why it matters when buying:** Large context windows have made 'just paste everything in' a genuine alternative to retrieval for small corpora. They have not made it a cheap one — you pay for every token on every call. More: https://bitfumes.com/answers/rag-vs-fine-tuning ### Prompt (also: system prompt, prompt engineering) A prompt is the text sent to a model to elicit a response; the system prompt is the standing part of it that sets role, rules and output format for every call. Prompt engineering is the practice of shaping that text until the behaviour is reliable. **Why it matters when buying:** Prompting is the cheapest lever and the first one to exhaust. A team that proposes fine-tuning before it has a written, version-controlled prompt and an evaluation set is reaching for the expensive tool first. More: https://bitfumes.com/answers/rag-vs-fine-tuning ### Fine-tuning Fine-tuning continues training an existing model on your own examples so it internalises a behaviour — a format, a tone, a classification scheme, a reasoning style. It changes how the model responds, not what facts it knows. **Why it matters when buying:** Fine-tuning is the right answer to a behaviour problem and the wrong answer to a knowledge problem, which is the single most expensive mix-up in applied AI. Knowledge that changes weekly cannot live in weights that are retrained quarterly. More: https://bitfumes.com/answers/rag-vs-fine-tuning ### AI agent (also: agentic AI, autonomous agent) An AI agent is a system where a model chooses which actions to take and in what order, calling tools and reacting to their results, rather than following a fixed script written by a developer. The defining property is that the control flow is decided at run time by the model. **Why it matters when buying:** That property is exactly what makes agents useful and what makes them fail. Reliability compounds down the chain: 95% per-step accuracy is about 60% across ten steps, so the successful deployments are narrow, short-chained and reversible. More: https://bitfumes.com/answers/ai-agents-in-production ### Tool use (also: function calling, tool calling) Tool use is the mechanism by which a model requests that your code run a specific function with specific arguments — a database query, an API call, a calculation — and then continues with the result. The model never executes anything itself; it emits a structured request your system chooses whether to honour. **Why it matters when buying:** 'The model chooses whether to honour' is where safety lives. Every irreversible action — a payment, a delete, an outbound email — should sit behind a check your code enforces, not behind an instruction in a prompt. More: https://bitfumes.com/answers/ai-agents-in-production ### Model Context Protocol (MCP) (also: MCP) The Model Context Protocol is an open standard, published by Anthropic in November 2024, for connecting AI assistants to external tools and data sources through a common interface. It replaces bespoke per-integration glue with servers any compatible client can call. **Why it matters when buying:** It matters commercially because it lowers switching costs. Integrations written against MCP move between assistants; integrations written against one vendor's proprietary plugin format do not. ### Multimodal model A multimodal model accepts or produces more than one kind of data — typically text plus images, audio or video — within the same request. In business use the common case is reading a document, screenshot or photograph and returning text. **Why it matters when buying:** This is what makes document-heavy back-office work automatable without a separate OCR pipeline, which is why invoice, form and claim intake are among the most reliably profitable AI projects. More: https://bitfumes.com/answers/ai-automation-for-mid-market-operations ### Structured output (also: JSON mode, constrained decoding) Structured output is a model feature that forces a response to conform to a supplied schema, usually JSON, so downstream code can parse it without defensive string handling. Modern implementations constrain generation rather than asking politely in the prompt. **Why it matters when buying:** It is the difference between a demo and a system. If a proposal has a model returning prose that a regular expression then scrapes, the integration will break the first time the phrasing changes. ## Retrieval & knowledge ### Retrieval-augmented generation (RAG) (also: RAG) Retrieval-augmented generation is a technique that fetches relevant passages from your own documents at query time and supplies them to the model as context, so the answer is grounded in your data instead of the model's training. The model's weights are untouched; only what it is shown changes. **Why it matters when buying:** Because nothing is baked in, a RAG system reflects a document edited five minutes ago. That property — not accuracy — is usually the reason it beats fine-tuning for business knowledge. More: https://bitfumes.com/answers/rag-vs-fine-tuning ### Embedding (also: vector embedding) An embedding is a list of numbers representing a piece of text, image or audio, positioned so that things with similar meaning sit close together in that numeric space. Comparing embeddings is how a system finds passages that are relevant without sharing keywords. **Why it matters when buying:** Similar is not the same as correct. Semantic search will confidently return the closest passage even when no passage in the corpus answers the question, which is why hybrid search and re-ranking exist. ### Vector database A vector database stores embeddings and finds the nearest ones to a query embedding quickly, at scale. Dedicated products exist, and mature general databases — PostgreSQL with pgvector, for example — do the same job. **Why it matters when buying:** Vector storage is rarely the hard part or the expensive part of a retrieval system. A proposal whose centrepiece is the choice of vector database is usually skipping past the parts that actually determine answer quality. ### Chunking Chunking is the process of splitting documents into passages small enough to retrieve precisely and large enough to remain self-contained. Strategy ranges from fixed token counts to splitting on document structure such as headings, clauses or table rows. **Why it matters when buying:** Chunking decides the ceiling on retrieval quality before any model is involved. If the answer to a question is split across two chunks and neither is retrievable on its own, no amount of model capability recovers it. ### Hybrid search Hybrid search combines keyword search with semantic vector search and merges the two result sets. Keyword search catches exact identifiers — part numbers, error codes, names — that embeddings blur together; semantic search catches paraphrases that keywords miss. **Why it matters when buying:** Almost every retrieval system that is 'mostly good but strangely bad at specifics' is missing the keyword half. It is a cheap fix and an early one. ### Re-ranking Re-ranking takes the passages a first search returned and reorders them with a slower, more accurate model that scores each one against the query directly. The usual pattern is to retrieve twenty or fifty candidates cheaply, then re-rank and keep the best few. **Why it matters when buying:** It is often the highest-return change available in a struggling RAG system, because it improves what the model is shown without touching the model or the corpus. ### Grounding Grounding is the practice of tying a model's output to specific supplied source material, and usually of citing which passage supported which claim. A grounded system can be checked; an ungrounded one can only be trusted or not. **Why it matters when buying:** Citations are the cheapest form of quality assurance you will ever ship. They let a subject-matter expert audit a hundred answers in an afternoon instead of accepting or rejecting the system wholesale. ### Knowledge cutoff The knowledge cutoff is the date after which a model's training data ends, so it has no inherent knowledge of later events. Retrieval, tool use and web access are the ways a system supplies what the model does not have. ## Quality & evaluation ### Hallucination A hallucination is a fluent, confident model output that is factually wrong or unsupported by the sources provided. It is a property of how these models generate text, not a bug that a future version removes entirely. **Why it matters when buying:** Because it cannot be eliminated, the design question is never 'will it be wrong' but 'what happens when it is'. Systems that survive contact with production put a reversible action or a human between a wrong answer and a consequence. More: https://bitfumes.com/answers/ai-agents-in-production ### Evaluation set (also: eval, golden dataset) An evaluation set is a fixed collection of representative inputs with known-good outputs, run against the system on every change to measure whether quality moved. It is the regression test suite of an AI system. **Why it matters when buying:** It is the single clearest signal of whether a team has shipped AI before. Without one, every prompt change is a guess, and 'it seems better' is the only available verdict. More: https://bitfumes.com/answers/how-to-hire-an-ai-consultancy ### LLM-as-judge LLM-as-judge uses a language model to score another model's outputs against a rubric, making large evaluation sets affordable to run. It is calibrated by checking its scores against human judgements on a sample. **Why it matters when buying:** Useful, and routinely oversold. An unvalidated judge measures agreement with itself, so ask how the judge was checked against humans before believing a quality number produced by one. ### Drift Drift is the degradation of a system's quality over time as the world, the inputs or the underlying model change while the system stays the same. Provider model updates are a common and often unannounced cause. **Why it matters when buying:** It is the reason an AI system is an operating cost and not a capital project. Budget for someone to own it after launch, or plan to discover the degradation from a customer. ### Observability (also: tracing) Observability for an AI system means recording every call — the prompt, the retrieved context, the model version, the output, the cost and the latency — so a specific bad answer can be reconstructed afterwards. Standard application logging does not capture enough to do this. **Why it matters when buying:** Without it, debugging a complaint about one wrong answer three weeks ago is impossible in principle rather than merely difficult. ### Human in the loop A human-in-the-loop design routes the model's output to a person for approval, correction or escalation before it takes effect. The model drafts; a human commits. **Why it matters when buying:** This is what makes an AI feature deployable in a regulated or high-consequence process years before full automation would be defensible — and the review data it produces is the training signal for narrowing the loop later. ## Cost & performance ### Inference cost Inference cost is what you pay each time the model runs, billed per input and output token at rates that differ between the two. It scales with usage, unlike the fixed cost of building the system. **Why it matters when buying:** It is the line item most often missing from an AI proposal. Ask for cost per transaction at your real volume, not a monthly total at a volume nobody has committed to. More: https://bitfumes.com/answers/how-to-reduce-ai-costs ### Prompt caching Prompt caching lets a provider reuse the processed form of a repeated prefix — a long system prompt, a fixed document set — across calls, charging a reduced rate for the cached portion and returning it faster. The savings depend on structuring prompts so the stable part comes first. **Why it matters when buying:** It is one of the few optimisations that cuts cost and latency at once, and it is frequently left on the table because it requires arranging the prompt deliberately rather than concatenating it. More: https://bitfumes.com/answers/how-to-reduce-ai-costs ### Distillation Distillation trains a smaller, cheaper model to reproduce the behaviour of a larger one on a specific task, using the large model's outputs as training examples. The result is narrower but much faster and cheaper to run. **Why it matters when buying:** It is the standard route out of a cost problem once a feature's scope has stopped moving — and the reason to prove a use case on a frontier model first rather than optimising prematurely. More: https://bitfumes.com/answers/how-to-reduce-ai-costs ### Open-weight model (also: open-source model, self-hosted model) An open-weight model is one whose trained parameters are published, so it can be downloaded and run on infrastructure you control. This is distinct from open-source in the software-licence sense; the weights are available, the training data usually is not. **Why it matters when buying:** The reason to choose one is almost always control — data residency, regulation, or removing a vendor dependency — rather than cost. Self-hosting trades a per-token bill for GPU capacity and an operations burden. More: https://bitfumes.com/answers/which-ai-model-should-we-use ### Frontier model A frontier model is one of the most capable models available at a given moment, typically from a major lab and typically the most expensive tier on offer. The label moves as the field does. **Why it matters when buying:** Use one to find out whether a use case is possible at all, then move down to the cheapest model that still passes your evaluation set. Proving feasibility and optimising cost are separate jobs and should not be attempted in the same week. More: https://bitfumes.com/answers/which-ai-model-should-we-use ### Latency (also: p95 latency, time to first token) Latency is how long a request takes to answer; for language models it is usually reported both as time to first token and as total completion time, and measured at a percentile such as p95 rather than as an average. A p95 of four seconds means one call in twenty is slower than that. **Why it matters when buying:** Averages hide the calls that make users abandon a feature. Agree the percentile before the build, because it constrains model choice, retrieval depth and whether responses can stream. ## Security & governance ### Prompt injection Prompt injection is an attack in which instructions hidden inside content the model reads — a web page, an email, a PDF, a support ticket — are followed as if they came from the user. It exists because a model sees instructions and data as the same stream of text. **Why it matters when buying:** It is not fully solvable by prompting, so it is contained architecturally: restrict what the model can reach, treat retrieved content as untrusted, and gate every consequential action behind code the model cannot talk its way past. More: https://bitfumes.com/answers/ai-agents-in-production ### Least-privilege retrieval Least-privilege retrieval means an AI system can only retrieve documents the requesting user is already entitled to see, enforced by the same permissions as the rest of your stack. The alternative — one index with one service account — collapses every access boundary in the organisation. **Why it matters when buying:** This, and not the model provider, is the most common real data breach in AI projects. It presents as a search feature cheerfully surfacing the salary spreadsheet. More: https://bitfumes.com/answers/is-our-data-safe-with-an-ai-consultancy ### No-training clause A no-training clause is a contractual commitment that data you submit will not be used to train the provider's models. Enterprise and API tiers of the major providers offer this by default; consumer tiers often do not. **Why it matters when buying:** Ask for it to extend to every subprocessor, not just the party you signed with. A consultancy passing your data to a model provider is a chain, and the guarantee is only as good as its weakest link. More: https://bitfumes.com/answers/is-our-data-safe-with-an-ai-consultancy ### Data residency Data residency is the requirement that data be stored and processed within a named jurisdiction. For AI systems it must cover inference too, not just storage, since a model call sends your data wherever the endpoint lives. **Why it matters when buying:** The storage half is usually handled and the inference half usually is not. It is worth asking about specifically rather than accepting a general assurance about hosting region. More: https://bitfumes.com/answers/is-our-data-safe-with-an-ai-consultancy ### Data processing agreement (DPA) (also: DPA) A data processing agreement is the contract governing how a processor handles personal data on a controller's behalf, required under GDPR and equivalent regimes. In an AI engagement it should name model providers as subprocessors, because they are processing on your behalf too. More: https://bitfumes.com/answers/is-our-data-safe-with-an-ai-consultancy ## How the work is bought ### Forward deployed engineer (FDE) (also: FDE, embedded engineer) A forward deployed engineer is a software engineer who works inside a customer's environment — their systems, their data, their constraints — building and shipping software for that specific customer rather than a general product. The term was popularised by Palantir and has since become the standard label for the model applied AI consultancies use. **Why it matters when buying:** The distinction from ordinary consulting is who holds the keyboard. An FDE engagement produces running code in your repository; an advisory engagement produces a document about code somebody else will write. More: https://bitfumes.com/answers/what-is-a-forward-deployed-engineer ### AI readiness (also: AI readiness assessment, AI opportunity assessment) AI readiness is the extent to which an organisation's data, processes and access controls would let an AI system be built and run — as distinct from whether it has a use case. A readiness assessment measures that, and normally identifies which candidate use cases pay back. **Why it matters when buying:** Readiness is mostly a data-access question and only slightly a technology one. The blocker is usually that nobody can approve the export, not that the data does not exist. More: https://bitfumes.com/answers/how-much-does-an-ai-consultant-cost ### Proof of concept (POC) (also: POC, pilot) A proof of concept is a time-boxed build whose only goal is to establish whether an approach works on real data, with no expectation that the code survives. A pilot is the next stage: the same capability run by real users on a limited scope, in production conditions. **Why it matters when buying:** Most stalled AI programmes are stuck between the two, having proven something in a notebook that was never going to survive access controls, latency budgets or edge cases. Agree at the start what evidence would justify moving on, and what would justify stopping. More: https://bitfumes.com/answers/how-long-does-it-take-to-build-an-ai-system ### Payback period The payback period is how long a project takes to return its total cost — build plus running costs — in measured savings or additional revenue. For AI work it is normally quoted in months and should include inference spend and ongoing maintenance. **Why it matters when buying:** Insist that the arithmetic is written down before the build, with a named metric and a current baseline. A project with no baseline cannot be shown to have worked, which is how AI budgets quietly disappear. More: https://bitfumes.com/answers/how-to-measure-ai-roi --- Source: https://bitfumes.com/answers/glossary