TL;DR — Track cost per coaching session as a first-class metric. It ties model spend directly to product value, surfaces regressions immediately when prompts or models change, and forces hard conversations about pricing and packaging early instead of after launch.
AI product unit economics are invisible without instrumentation. The bill arrives monthly; the decisions get made every day. Closing that loop — per-session cost as a first-class telemetry signal — is what makes AI product pricing defensible.
What we capture per session
- Input tokens, output tokens, cache hit tokens (separately)
- Model used per turn (we have multiple models in Blink AI for different turn types)
- Embedding tokens for memory writes
- Vector store query count + latency
- Total session duration and turn count
- Tenant + user (for tenant-level analytics; no PII in the cost log)
The metrics we look at
- Median cost per session — the typical case
- P95 cost per session — the long-tail sessions that drag the average
- Cost trend — week over week, by tenant cohort
- Cache hit rate — should grow over time as prompts stabilize
- Cost per quality unit — pair cost with rubric score from evaluation pipeline
Why this metric drives everything
- Pricing — if cost per session is $0.40, you cannot sell sessions for $0.30
- Packaging — high-value tenants might warrant frontier models; light-touch tenants run on cheaper routing
- Performance work — what optimization buys back enough to be worth the time?
- Regression detection — prompt change cost the median session 30% more tokens? Flag immediately.
The trap
Optimizing cost without watching quality. Easy to cut tokens. Easy to do it badly. Always pair the cost dashboard with a quality dashboard. The session that costs 30% less but scores 20% lower on coaching rubric is not a win.
Frequently asked questions
Why per-session vs per-token?
Sessions are what users do; tokens are what the model bills. Per-session ties cost directly to product value. Per-token is the input to per-session, but it is not the unit of decision-making.
What if cost is too high?
Several levers (see optimization): smaller model for routing, shorter context, prompt caching, batching. Decide based on quality impact, not just cost.
Working on something similar?
T-Square architects, builds and operates production systems for learning, AI and custom software products. Talk to a senior engineer for a second opinion.