TL;DR — The LLM is stateless. Your application is not. The structured state your orchestrator maintains between turns matters more than the prompt — it is the difference between a chat that remembers and one that drifts.
Every multi-turn AI product solves the same problem: the model has no memory, and yet the product has to behave like it does. The naive answer is “put the history in the prompt”. The grown-up answer is to design the structured state your orchestrator maintains and to re-inject it on every turn.
Three kinds of state
- Recent turn buffer — last N exchanges, kept verbatim. The conversational glue.
- Structured session state — declared goal, phase, commitments, open questions. Application-controlled, model-readable.
- Long-term memory — embeddings of past sessions, retrieved on demand (per-user, isolated)
The orchestrator pattern
- On every turn, load the session state from the database
- Decide which long-term memories are relevant (vector retrieval)
- Build the system prompt: persona + session state summary + retrieved memories + instructions
- Call the model with system + recent turns + new user input
- Parse the response — extract any state updates (new commitment, phase change, surfaced insight)
- Persist the updated session state. The next turn starts here.
Why this beats prompt-stuffing
- Token cost stays roughly flat per turn instead of growing
- Prompt caching works (the system prompt is stable; only the new input changes)
- State is queryable and observable outside the prompt
- You can change models without losing conversation continuity
- State survives session restarts, app reloads, model upgrades
Designing the state shape
Keep it small, keep it structured. Each field has a name the model can recognize when re-injected. Avoid free-form blobs — they read back as noise. Blink AI's state carries phase, declared goal, commitments and an insight log — four fields, each disciplined.
Frequently asked questions
Why not put everything in the prompt?
Tokens cost money and context degrades quality past a certain length. Structured state outside the prompt is cheaper, faster, and the model sees exactly what you intended — not whatever survived the truncation window.
What goes in structured state vs the prompt?
Stable facts (declared goal, commitments, phase) live in structured state and get re-injected every turn. Recent turns live in the prompt. Long-term memories live in a vector store and get retrieved.
Working on something similar?
T-Square architects, builds and operates production systems for learning, AI and custom software products. Talk to a senior engineer for a second opinion.