Labeled Recall: Information-Flow Control for Memory

The through-line of safe agent memory is a single idea applied consistently: every stored item carries a label, and the label governs what the item is allowed to influence. Where did it come from, how much can it be trusted, when was it written, and who does it belong to? With those labels attached and propagated, a runtime can make deterministic decisions about recall instead of hoping the model behaves. This is information-flow control applied to memory, and it is what finally turns a store from an open attack surface into a governed one.

Every memory carries a label

A raw memory value is not enough. Alongside the content, store its provenance: the source it came from, a trust level, a timestamp for freshness, and the owner or scope it belongs to. Those four labels are what let later decisions be principled rather than guesswork.

Without them, every recall is anonymous, and an anonymous recall is one you cannot safely reason about. With them, an item's history travels with it wherever it goes.

Every memory item carries its source, trust level, freshness, and owner.

Labels travel with data

A label is only useful if it propagates. When a memory is combined, summarized, or used to derive something new, the labels come along: anything derived from an untrusted source stays untrusted, and anything derived from a confidential item stays confidential.

This propagation is what closes the gap that memory poisoning exploits. Untrusted origin cannot be laundered into trusted knowledge simply by passing through a few steps, because the label follows the data.

Check the label before acting

With labels in place, recall gains a checkpoint. Before a memory is allowed to influence a consequential action, the runtime inspects its label and applies policy: a low-trust or stale item may inform reasoning but not trigger an action; a confidential item may not flow to an incompatible destination.

The decision is deterministic and made by the runtime, not left to the model's discretion. That is the whole point: safety by a rule, not by a hope.

A recalled memory's label is checked against policy before it can influence an action.

The two rules that matter

Most of the value comes from two simple, deterministic rules. First, untrusted data may never trigger a consequential action; it can be read and reasoned about, but not turned into an effect. Second, data may only flow to destinations compatible with its confidentiality label, so sensitive memory cannot leak outward.

Placed on a grid of trust against consequence, the only cell that must be blocked outright is untrusted memory driving a high-impact action. Everything else can proceed under its appropriate handling.

The one cell to block is untrusted memory driving a high-impact action.

Deterministic, not hopeful

The reason to enforce recall with labels rather than instructions is the same reason the whole harness exists: a prompt asking the model to ignore untrusted memory is a request a clever input can undo, while a label check is a control it cannot.

Deterministic information-flow control does not replace good judgment; it guarantees the floor beneath it. No matter how the model reasons, an untrusted memory simply cannot become a consequential action.

✅

A floor you can trust. Label checks guarantee that untrusted memory cannot drive a consequential action, whatever the model decides.

Bringing memory together

Labeled recall is where this series converges. The three memory types tell you what kind of thing you are storing. The context graph connects it. Treating memory as an attack surface tells you why it needs defending. The promotion pipeline decides what earns trust. And labeled recall enforces, at the moment of use, what each item is allowed to do.

Together they turn memory from the agent's greatest liability into its most useful, and governable, asset.

What a practitioner decides

Adopting labeled recall means deciding the label schema, how labels propagate through derivation, and the deterministic rules that govern what a labeled memory may influence or where it may flow. These are explicit, testable policies, not prompt suggestions.

The question for any recalled memory is no longer just what does it say. It is: what is its label, and what does that label permit it to do?

Key takeaways

Attach source, trust, freshness, and owner labels to every memory item.
Propagate labels through derivation so untrusted origin can't be laundered into trust.
Check a memory's label against policy before it influences a consequential action.
Enforce two rules: untrusted data can't trigger action; data flows only to compatible destinations.
Block the one cell that matters: untrusted memory driving a high-impact action.
A label check is a control a clever input cannot undo, unlike a prompt request.

Glossary

Labeled recall: Attaching governing labels to memory and checking them before an item can influence an action.
Information-flow control: Labeling data, propagating the labels, and checking them before acting or egressing.
Provenance: The recorded origin of a piece of data.
Label propagation: Carrying a data item's labels onto anything derived from it.
Consequential action: An action with real effect that untrusted data must not be allowed to trigger.
Confidentiality label: A marker constraining which destinations a piece of data may flow to.
Policy check: A deterministic runtime decision about what a labeled item is permitted to do.