Get in touch

TL;DR — Six techniques cut our LLM bill 40-60% without dropping coaching quality: prompt caching, smaller router models, summarization of recent history, structured outputs, batch processing for non-real-time work, and aggressive retry-with-cheaper-model.

Six LLM cost levers — what each one buysPrompt caching is the biggest single win. Combined with model routing and structured outputs, 40-60% reductions are common. — /cost levers 1. Prompt caching−80% input cost (cache hit) 2. Smaller routing models−95% on routing turns 3. Summarize history−40% context tokens 4. Structured outputs−20-40% output tokens 5. Batch processing−50% non-realtime 6. Cheap-first retry−70% on routine tasks Combined: 40-60% bill reduction at unchanged quality ⚠ always pair cost dashboard with quality dashboard
Prompt caching is the biggest single win. Combined with model routing and structured outputs, 40-60% reductions are common.

Frontier LLMs are powerful and expensive. The temptation is to use one model for everything. The discipline is to figure out which turns actually need the frontier model and which can run on something cheaper, and to make every token earn its place in the prompt.

The six techniques

1. Prompt caching

Stabilize the system prompt and the first portion of context. Cache hit cuts input cost by 80-90%. Requires resisting the temptation to dynamically rebuild the system prompt on every turn.

2. Smaller models for routing

Classify the user's intent with a 7-8B parameter model. Route the actual response generation to a frontier model only when the intent warrants it. ~95% of routing accuracy at <5% of frontier cost.

3. Summarize recent history

Past 30 turns? Keep last 5 verbatim, summarize the rest. The summary lives in structured session state and gets re-injected — much cheaper than re-sending raw history.

4. Structured outputs

JSON mode or function calling forces concise responses. The model stops padding answers. Output tokens drop 20-40%.

5. Batch processing

For evaluation runs, content moderation, embedding generation — anything not real-time — use the batch API. 50% cheaper, 24-hour SLA.

6. Retry with a cheaper model first

For deterministic tasks (classification, extraction), try the smaller model first. Escalate to the frontier model only on confidence-below-threshold. Most of the time the cheaper model is right.

What we do not recommend

  • Switching to open-source models without an evaluation pipeline — quality drops are silent
  • Aggressive prompt compression — easy to lose meaning, hard to detect
  • Cost-only optimization — track quality alongside cost on every change

Frequently asked questions

Does prompt caching really help?

Yes — for stable system prompts, prompt caching can cut input token cost by 80-90% on cache hits. The investment is making the cached portion of the prompt actually stable, not regenerating it per request.

Smaller model for routing — how much smaller?

For Blink AI we use a 7B-parameter model to classify user intent and route the actual coaching turn to a frontier model. The router catches ~95% of intent correctly at <5% of the frontier-model cost.

Working on something similar?

T-Square architects, builds and operates production systems for learning, AI and custom software products. Talk to a senior engineer for a second opinion.