Memory & State Across Turns — Context Engineering & Long-Context

A single chat message is easy to understand: a person asks, a model answers. Real conversations are harder because meaning accumulates. “That customer,” “the second option,” and “do it again but shorter” only make sense if the system can connect the current turn to earlier ones. Memory and state are the engineering practices that make those connections possible. They are also where many assistant failures begin: stale facts, hidden assumptions, leaked data, and confident answers based on the wrong past.

A turn is not the whole conversation

A turn is one exchange in a conversation, usually a user message followed by an assistant response. Large language models do not naturally “remember” earlier turns the way a person does. At inference time, the model receives an input and produces an output. If earlier context is not included in that input, the model cannot reliably use it.

What feels like memory is usually prompt construction. The application gathers the current user message, selected prior messages, system instructions, tool results, user profile data, and other relevant facts, then sends that assembled prompt to the model. The model sees only what fits into that prompt at that moment.

This distinction matters because memory is not magic and not free. Every remembered item must be stored somewhere, retrieved somehow, judged relevant, and placed into the model’s limited context window. Good memory design is therefore less about saving everything and more about deciding what deserves to travel forward.

State is the system’s working record

State is the information the application keeps about what is happening now. It may include the conversation identifier, the authenticated user, the current task, chosen settings, tool outputs, approvals, pending actions, and references to business records. State is often structured, explicit, and owned by the application rather than by the model.

For example, in a travel assistant, state might say that the user is comparing flights from Seattle to Boston, prefers morning departures, has not yet approved payment, and is currently looking at option three. The model can help discuss that state, but the application should be the source of truth for it.

Keeping state outside the model gives engineers control. They can validate it, log it, expire it, encrypt it, and test it. They can also prevent the model from treating casual language as authorization. If a user says “book the usual,” the application state should still require the right identity, policy checks, and explicit confirmation before any real transaction occurs.

📌

Plain rule. Let the model reason over state, but do not make the model the only place where important state exists.

The application owns validated state as the source of truth; the model can discuss it but should not be treated as authoritative.

Memory comes in different kinds

Memory is broader than state. It is any information from the past that may help future turns. The most basic kind is conversation history: recent messages carried forward so the assistant can resolve pronouns, follow instructions, and avoid asking the same question twice.

A second kind is summary memory. Instead of carrying every prior message, the system periodically writes a compact summary: decisions made, preferences stated, open questions, and constraints. Summary memory saves space, but it can lose nuance or accidentally turn uncertain statements into facts.

A third kind is long-term memory. This stores information beyond one session, such as a user’s preferred tone, recurring business context, or previously approved project details. Long-term memory is powerful because it makes the assistant feel continuous, but it also raises higher expectations for consent, correction, retention, and security.

A fourth kind is retrieved knowledge. The assistant may search documents, tickets, emails, records, or databases and bring back relevant passages for a turn. This is often called retrieval-augmented generation, meaning the model’s answer is augmented by information retrieved from an external source.

Use recent turns for local coherence.
Use summaries for long conversations.
Use long-term memory only for facts worth keeping.
Use retrieval when the answer depends on governed external sources.

Memory ranges from recent conversation history to compact summaries to long-term facts that persist across sessions.

The context window is a budget, not a filing cabinet

The context window is the maximum amount of text and other encoded input the model can consider at one time. It is tempting to treat a large context window as permission to paste everything into the prompt. That usually makes systems slower, more expensive, and sometimes less accurate.

A model must pay attention across everything it receives. Irrelevant text can distract it, create contradictions, or make the answer depend on accidental details. More context is not always better context. The real goal is to include the smallest set of information that is sufficient for the current task.

This creates an engineering discipline: rank candidate memories, filter by relevance, remove duplicates, prefer authoritative sources, and keep instructions separate from data. When there is a conflict, the system should know which source wins. For example, a current policy document should override a year-old conversation summary.

⚠️

Context clutter. Stuffing the prompt with every available memory can make an assistant less reliable, not more informed.

A healthy memory loop has gates

Memory should be managed as a loop, not as a heap. The assistant observes a turn, decides whether anything is worth remembering, writes only approved items, retrieves selected items later, and uses feedback to correct or delete what is wrong. Each step is a chance to improve quality and reduce risk.

The most important gate is the write gate. Not every sentence should become memory. “I like short answers” may be useful. “My passport number is...” is sensitive and should normally be protected, minimized, or not stored at all unless the application has a clear reason and the right safeguards.

The second gate is the read gate. Even if a fact is stored, it should not be used in every turn. A user’s dietary preference may matter in meal planning and be irrelevant in a budget review. Good systems retrieve memory by purpose, permission, freshness, and relevance.

Manage memory as a loop: observe each turn, gate what is written, retrieve by purpose, and correct or delete what is wrong.

Safety, privacy, and identity shape memory

Memory can improve an assistant, but it can also preserve mistakes and expose sensitive information. A memory system should answer basic questions plainly: What is stored? Why is it stored? Who can read it? How long does it live? How can it be corrected or deleted?

Identity is central. If the application uses Microsoft Entra ID for sign-in and access control, memory should be partitioned according to the same identity and authorization model. A user should not retrieve another user’s private conversation summary simply because both users asked similar questions.

Enterprise systems also need tenant, role, and data-boundary awareness. A memory written while a user had access to a project may become unsafe after that access is removed. Retrieval should check current permissions, not just historical permissions. The safest pattern is to store references to governed records where possible and re-check access when those records are used.

Tools add another risk. If an assistant can call Foundry Tools, search data, send email, or update systems, remembered instructions must not silently become action authority. Memory can suggest context; it should not replace authentication, authorization, policy, and user confirmation.

Partition memory by user, tenant, and role.
Re-check permissions at retrieval time.
Minimize sensitive personal data in stored memory.
Keep audit records for memory writes, reads, corrections, and deletions.
Require explicit confirmation before high-impact actions.

⚠️

Stale access. A memory that was safe yesterday can become unsafe after roles, projects, or permissions change.

Retrieval should check current permissions not historical ones, so a memory becomes unusable once a user loses access.

Design choices leaders and practitioners must make

Memory is not only a model feature; it is a product, security, and operations decision. Leaders must decide how personal the assistant should feel, how much risk the organization will accept, and which memories are worth the cost of governing them. Practitioners must turn those decisions into defaults, controls, tests, and observability.

A customer-support assistant may need short-lived case context, strict access checks, and complete auditability. A personal productivity assistant may benefit from long-term preferences, but it needs clear user controls. A software engineering assistant may need repository-specific memory that disappears when a contributor leaves a project. The right memory design follows the work, not the trend.

Microsoft Foundry can help teams build and evaluate model applications, but the architecture still needs explicit decisions about storage, retrieval, identity, retention, and human control. The platform does not remove the need to define what the assistant is allowed to know and when it is allowed to use that knowledge.

Test memory like a product surface

A memory feature should be tested directly, not inferred from a few good conversations. Tests should check whether the assistant remembers useful facts, forgets forbidden facts, respects corrections, handles contradictions, and avoids using memory in the wrong context.

Useful test cases include time passing, permissions changing, users sharing devices, similar users asking similar questions, and old preferences conflicting with new instructions. The question is not only “Did the model answer well?” but also “Did the system choose the right past to bring forward?”

Observability matters after release. Teams should be able to inspect why a memory was written, why it was retrieved, what prompt it entered, and how users corrected it. Without that trail, memory becomes folklore: everyone can feel its effects, but no one can explain or improve them.

✅

Good sign. A mature memory system can explain what it remembered, why it used it, and how a person can change it.

Key takeaways

Models do not automatically remember earlier turns; applications must supply selected context at each turn.
State is the application’s working record and should remain explicit, validated, and governed.
Memory types include recent history, summaries, long-term facts, and retrieved knowledge, each with different risks.
The context window is a limited budget, so relevance and authority matter more than volume.
Memory needs gates for writing, reading, correction, retention, and deletion.
Identity, permissions, and auditability are core parts of memory design, not afterthoughts.

Glossary

Turn: One exchange in a conversation, usually a user message and the assistant response that follows.
State: The structured working information an application keeps about the current user, task, settings, approvals, and process.
Memory: Information from earlier interactions or sources that may be used to improve later turns.
Prompt construction: The process of assembling instructions, user input, memory, tool results, and other context before sending a request to a model.
Context window: The maximum amount of input a model can consider at one time when generating a response.
Retrieval-augmented generation: A pattern where an application retrieves relevant external information and includes it in the model’s input to ground the answer.
Authorization: The process of deciding whether an authenticated person or system is allowed to access data or perform an action.
Retention: The policy and practice that determine how long stored information is kept before it is deleted or archived.