Get in touch

TL;DR — The LLM is stateless. Your application is not. The structured state your orchestrator maintains between turns matters more than the prompt — it is the difference between a chat that remembers and one that drifts.

Three layers of conversation state in AI productsRecent turns in the prompt · structured session state re-injected each turn · long-term memory retrieved on demand. — /state layers layer 1 Recent buffer last N turns verbatim ~5 turns in prompt conversational glue layer 2 Session state structured + persisted phase: GOAL goal: “ship feature X” commitments: 2 insights: [..] re-injected every turn layer 3 Long-term memory vector store ∞ embeddings retrieved per-user isolated
Recent turns in the prompt · structured session state re-injected each turn · long-term memory retrieved on demand.

Every multi-turn AI product solves the same problem: the model has no memory, and yet the product has to behave like it does. The naive answer is “put the history in the prompt”. The grown-up answer is to design the structured state your orchestrator maintains and to re-inject it on every turn.

Three kinds of state

  • Recent turn buffer — last N exchanges, kept verbatim. The conversational glue.
  • Structured session state — declared goal, phase, commitments, open questions. Application-controlled, model-readable.
  • Long-term memory — embeddings of past sessions, retrieved on demand (per-user, isolated)

The orchestrator pattern

  1. On every turn, load the session state from the database
  2. Decide which long-term memories are relevant (vector retrieval)
  3. Build the system prompt: persona + session state summary + retrieved memories + instructions
  4. Call the model with system + recent turns + new user input
  5. Parse the response — extract any state updates (new commitment, phase change, surfaced insight)
  6. Persist the updated session state. The next turn starts here.

Why this beats prompt-stuffing

  • Token cost stays roughly flat per turn instead of growing
  • Prompt caching works (the system prompt is stable; only the new input changes)
  • State is queryable and observable outside the prompt
  • You can change models without losing conversation continuity
  • State survives session restarts, app reloads, model upgrades

Designing the state shape

Keep it small, keep it structured. Each field has a name the model can recognize when re-injected. Avoid free-form blobs — they read back as noise. Blink AI's state carries phase, declared goal, commitments and an insight log — four fields, each disciplined.

Frequently asked questions

Why not put everything in the prompt?

Tokens cost money and context degrades quality past a certain length. Structured state outside the prompt is cheaper, faster, and the model sees exactly what you intended — not whatever survived the truncation window.

What goes in structured state vs the prompt?

Stable facts (declared goal, commitments, phase) live in structured state and get re-injected every turn. Recent turns live in the prompt. Long-term memories live in a vector store and get retrieved.

Working on something similar?

T-Square architects, builds and operates production systems for learning, AI and custom software products. Talk to a senior engineer for a second opinion.