10. Layer 5: Memory & State

Problem Solved

Context is finite, but tasks may require information across steps, sessions, or runs. Layer 5 separates what the model sees now from what the system remembers for later.

Service Provided to the Layer Above

Memory and state provide relevant information to the control loop without forcing every fact into the active context window.

Memory Tiers

Tier	Lifetime	Storage	Example
Short-term memory	Current run	Active context	System prompt, recent steps, latest observation
Working state	Current task	Harness state store	Counters, current plan, file references, intermediate results
Long-term memory	Across runs	External database, vector index, key-value store (20)(21)	User preferences, past task summaries, domain facts
Artifact store	Current or long-lived	Object storage, filesystem, database blobs	Large documents, images, generated reports, datasets

The key distinction:

Context is what the model sees. Memory is what the system can choose to show the model.

The context window has a hidden cost. Transformer models compare every token in the context to every other token, and that work grows quadratically with length: the per-layer attention cost is O(n²·d), so doubling the context roughly quadruples the attention work (22). A long context is therefore not just expensive per token, each extra token also makes every other token costlier to process. Keeping the active context small and retrieving only what is needed is a direct latency and cost optimization, not just a budgeting choice.

State Keys

Large objects should not be inlined into context.

Bad:
  Paste a 50-page document into every model call.

Good:
  Store the document externally.
  Keep a reference key in state.
  Retrieve only the relevant section when needed.

State keys let tools and model decisions refer to objects without bloating the context window.

State item	Context representation
Large file	File ID or path reference
Query result	Result ID plus short summary
Image or audio	Artifact ID plus metadata
Plan	Current step plus goal summary
Tool output	Structured summary plus reference to full output

Memory Operations

Operation	Purpose
Write	Store a fact, artifact, or state update
Retrieve	Fetch relevant information for the current decision
Summarize	Compress long history into decisions and constraints (11)
Supersede	Mark old state as replaced by newer state
Expire	Remove information that is no longer valid
Verify	Check source, timestamp, or permission before use

Design Rules

1. Do not treat the context window as a database.
2. Retrieve only information relevant to the next decision.
3. Keep timestamps and sources for stored facts.
4. Separate raw artifacts from summaries.
5. Make state updates deterministic and auditable.
6. Mark superseded information instead of silently mixing old and new state.
7. Protect sensitive memory with access controls.

Failure Modes

Failure	Symptom	Fix
Stale memory	Agent acts on outdated facts	Timestamps, supersession, refresh policies
Irrelevant retrieval	Context fills with unrelated facts	Better retrieval scope, ranking, filters
Contradictory memory	Old and new facts conflict	Supersede rules, explicit conflict resolution
Artifact bloat	Large objects consume context	State keys and on-demand retrieval
Memory leak	State grows without expiration	TTL, compression, archival
Trusting retrieved text	Retrieved content is treated as instruction (23)	Separate data from instructions, sanitize retrieved content