LLM Cost Optimisation: Cutting Spend Without Cutting Quality

What does LLM cost optimisation actually mean?

LLM cost optimisation is the practice of reducing the compute and token spend of a production AI system while holding output quality constant or improving it. It is not about swapping to the cheapest model and hoping nobody notices. Done well, it combines model selection, caching, batching, prompt discipline, and observability — each targeting a different part of the cost curve.

Most teams that overspend on LLMs made one architectural decision early — usually "call the flagship model for everything" — and never revisited it as usage scaled. The fix is rarely a single swap. It is a set of layered engineering decisions, each with a measurable, defensible saving.

How does model tiering cut costs without cutting quality?

Model tiering means routing each request to the cheapest model capable of handling it correctly, rather than sending every call to the most capable (and most expensive) model available. It is the single most direct lever for controlling LLM spend, because token pricing typically varies by an order of magnitude between a provider's fast, lightweight tier and its top-end reasoning tier.

Anthropic's own Claude family illustrates the pattern: lightweight models such as Claude Haiku are explicitly positioned for "fast responses for high-volume, latency-sensitive applications," while the higher-reasoning tiers are reserved for demanding, long-horizon agentic work. Anthropic's developer lifecycle guidance goes further and names cost optimisation as a formal stage of shipping to production — alongside evals and batch testing — not an afterthought bolted on once the bill arrives.

In practice, this means classifying your workload before you write a single prompt. Simple classification, extraction, summarisation of short text, and high-volume triage tasks are strong candidates for a cheaper, faster model. Multi-step reasoning, long-context synthesis, and agentic tool-use chains are where the premium tier earns its price. A well-designed system routes dynamically between tiers based on task complexity, not a single model chosen at project kickoff and never revisited.

What is prompt caching and when does it pay off?

Prompt caching is a mechanism that reduces cost and latency for repeated context — system prompts, tool definitions, retrieved documents, or few-shot examples that are sent unchanged across many calls. Instead of paying full input-token price every time, cached context is billed at a substantially reduced rate on subsequent calls.

Caching pays off fastest in applications with a large, stable context block and high call volume: RAG pipelines that re-send the same retrieved passages across a session, agents that re-send the same tool schema on every step, and chat applications with a long, unchanging system prompt. If your architecture leans on retrieval, it's worth reading our piece on RAG vs fine-tuning to understand how retrieved context volume interacts with caching economics.

Caching does not help workloads where context changes on every call — highly personalised prompts with little shared structure will see minimal benefit. Knowing which of your call patterns are cacheable is itself a useful audit exercise.

Does batching reduce LLM costs?

Batching processes multiple requests together rather than one at a time, which reduces per-call overhead and, on several providers, qualifies for discounted asynchronous pricing. It is most effective for workloads that don't require an immediate response — nightly enrichment jobs, bulk classification, backfilling historical records, or generating embeddings for a document corpus.

The trade-off is latency: batched jobs typically return results on the order of minutes to hours rather than seconds. That's an acceptable trade for back-office and data-pipeline work, but not for a customer-facing chat interface. Segmenting your workload into "needs to be synchronous" and "can tolerate delay" is a prerequisite for using batching properly.

What is a prompt budget, and why does it matter?

A prompt budget is an explicit limit on the input and output tokens a given call is allowed to consume, enforced at the application layer rather than left to whatever the model happens to generate. Without one, costs creep upward silently as context windows grow, retrieved documents accumulate, and output length drifts with model updates.

Practical prompt budgeting includes: trimming retrieved context to only the passages that materially inform the answer, summarising or truncating conversation history instead of resending it in full, and constraining output length where verbose responses add cost without adding value. None of this requires a new model — it requires treating token count as a first-class engineering metric, reviewed the same way you'd review query performance or API latency.

Why does observability matter for LLM spend?

LLM observability is the practice of tracking token usage, latency, cost per request, and output quality per model and per use case in production — not just in aggregate on a monthly invoice. Without it, cost optimisation is guesswork: you can't route intelligently between tiers, prove that a cheaper model maintains quality, or catch a prompt regression that silently doubles token usage.

At minimum, track cost and token count per endpoint or per feature, tag requests by model tier and task type, and run before/after evals whenever you change a model or a prompt template. This is also where cost optimisation intersects with reliability: a model swap that saves money but degrades output quality on 5% of requests is not a saving — it's a support ticket. If you haven't yet assessed whether your organisation has the monitoring maturity to do this safely, our AI readiness assessment piece is a useful starting point.

A worked example: tiering a support-triage pipeline (illustrative)

To show how these levers compound, consider a hypothetical support-triage pipeline that classifies incoming tickets, drafts a suggested response, and escalates complex cases to a human agent. The figures below are illustrative arithmetic, not measured client data — they show the mechanics of the saving, not a benchmarked outcome.

Assume a pipeline that originally sends every ticket to a top-tier reasoning model for both classification and drafting. An audit finds that roughly 80% of tickets are simple, high-volume classification tasks (routing, tagging, sentiment), while only 20% require the reasoning tier for genuinely ambiguous or multi-step cases.

Re-routing the 80% simple-classification volume to a lightweight, high-throughput tier — while keeping the reasoning tier for the harder 20% — means only a fifth of calls still incur premium-tier pricing. Layering prompt caching on top (the classification prompt and tool schema are identical across nearly every call) further reduces the input-token cost of that remaining volume, since repeated context is billed at a reduced cached rate rather than full price each time.

The combined effect, in this illustrative scenario, is a substantial reduction in total token spend against the "everything through the top-tier model" baseline — driven almost entirely by matching task complexity to model tier, plus caching the repeated structural context. The exact percentage will vary by provider, pricing tier, and cache hit rate in any real deployment, which is why this is presented as a worked mechanism rather than a promised outcome.

Lever	What it does	Effort to implement	Best suited to
Model tiering	Routes tasks to the cheapest capable model	Medium — needs task classification	High-volume, mixed-complexity workloads
Prompt caching	Discounts repeated context on subsequent calls	Low — mostly configuration	Stable system prompts, RAG, agent tool schemas
Batching	Processes requests asynchronously at lower unit cost	Low-Medium	Non-time-sensitive bulk jobs
Prompt budgets	Caps input/output token count per call	Low — policy plus code changes	Any workload with growing context or verbose output
Observability	Tracks cost, latency, quality per model/use case	Medium — requires tagging and dashboards	Any production LLM system, especially multi-tier

How do you avoid vendor lock-in while optimising cost?

Avoiding lock-in means building on a model-agnostic architecture so you can move workloads between providers as pricing or capability shifts, rather than being stuck on one vendor's pricing curve. Frameworks such as LangChain expose a standard model interface across providers including OpenAI, Anthropic, Gemini, Azure OpenAI, and Bedrock, so switching providers requires minimal code changes once your application logic is decoupled from any single API.

For organisations with data-sovereignty requirements or particularly high, predictable volume, running open-weight models locally via tools like Ollama is a genuine cost option worth evaluating alongside hosted APIs — it shifts the cost structure from per-token billing to infrastructure ownership, which changes the calculus at scale. This is also where data infrastructure maturity matters: portable, well-governed data pipelines make it far easier to benchmark models fairly, since you're comparing outputs on the same inputs rather than re-engineering context for each provider.

The broader point, and one we hold to across engagements, is that the model choice is rarely the biggest lever. The surrounding architecture — prompting discipline, retrieval quality, evaluation, and monitoring — typically matters more to production cost and quality than which model sits behind the API. A model-agnostic approach protects against lock-in and keeps the option open to pick the best model for each task as the market shifts.

Where cost engineering fits into a broader AI strategy

Cost optimisation is not a one-off audit — it's an ongoing discipline that should sit alongside your evaluation and monitoring practice from day one, not be retrofitted after the first surprising invoice. Teams building or scaling LLM applications benefit from establishing model tiering, caching, and observability as standard practice during initial architecture, covered in more depth in our AI engineering work and our approach to AI product strategy. For teams further back in the pipeline — still building the data foundations that make reliable model comparison possible — it's worth reviewing your data infrastructure before optimising the model layer itself, since inconsistent inputs make any cost/quality comparison unreliable. You can browse more practical guides like this on our insights page.

If you're exploring how to bring LLM costs under control without compromising the quality your product depends on, we can help — starting with an honest audit of where your spend is actually going.