AI agent memory: what it actually is, and the two senses

AI agent memory, defined

Model calls are stateless. OpenAI's docs say it plainly: if you want the model to see earlier turns, re-send them or point at a previous response ID. Anthropic documents the context window as a per-call budget — 200,000 tokens for most Claude models — not as storage. Weights do not change when a user tells an agent their deploy target is Fly.io.

So every product that appears to remember is running code around the model that wrote something down and put it back in the prompt later. That code is the memory system. Not the model.

Where the term came from

Two papers set the shape, both in 2023. Generative Agents (arXiv:2304.03442, April 2023) gave 25 simulated characters a "memory stream": an append-only log of observations, with retrieval scored on recency, importance and relevance. MemGPT (arXiv:2310.08560, October 2023) framed it as operating-system paging — a small "main context" the model sees, a large "external context" it does not, and function calls that move data between them.

The vocabulary was borrowed from cognitive psychology, and the borrowed terms map poorly onto the implementations they now name. "Short-term" and "long-term" imply decay; almost no production system implements decay, it implements a DELETE.

By February 2024 it was a consumer feature: OpenAI shipped memory in ChatGPT with user-visible, deletable entries. Frameworks then hardened the split — LangGraph separates a thread-scoped checkpointer from a namespaced cross-thread store, which is the cleanest formalisation of the two senses below.

The two senses people mix up

"Memory" is used for two different things, and most arguments about it are people using different senses.

When a buyer says "the agent remembers me," they mean Sense 2. When an engineer says "memory is handled," they usually mean Sense 1 — a message buffer with a summariser. Those are not the same product. A summariser that compresses a 40-turn thread loses everything the moment the session ends.

Say which one you mean in the first sentence.

What it is not

The context window is capacity for one call; memory is a decision about what enters that capacity. Stuffing the full history into a 1M-token window is not memory, and it does not even work as a shortcut — "Lost in the Middle" (arXiv:2307.03172) showed accuracy drops when the relevant fact sits mid-context.

RAG shares the retrieval plumbing and differs on the write path. RAG reads a corpus somebody else authored: docs, tickets, contracts. Memory reads facts the agent itself decided to persist. If nothing in your system chooses what to write, you have retrieval, not memory.

Agentic state is the scratchpad inside a single run — the plan, the tool results, the step counter — and it dies at the end of the run by design. See agentic state: the data an agent keeps between steps.

Fine-tuning changes weights and cannot be deleted per-user on request. Wrong tool for preferences, permanently.

The two paths that make it memory

Most teams build the read path and call it done. The write path is where memory actually lives, and it is the one nobody instruments.

A worked example

A support triage agent for a self-serve SaaS. One table in Postgres with pgvector for the index. Memory rows look like this:

{
  "id": "m_01H9",
  "namespace": "user:4417",
  "kind": "preference",
  "text": "Deploys to Fly.io; does not use Docker Compose locally.",
  "source_run": "run_2f8c",
  "confidence": 0.8,
  "created_at": "2025-03-04T11:20:07Z",
  "expires_at": "2025-06-02T11:20:07Z"
}

The write path runs after the response, not during it: a second cheap model call reads the turn and emits zero or more candidate facts, each tagged preference, identity, constraint or decision. Candidates are embedded, compared against existing rows in the same namespace, and either merged or dropped above a similarity threshold. Free-text transcripts are never written — a transcript is not a fact, and storing it just moves the context problem into Postgres. Ninety-day expiry on preferences, no expiry on identity.

The read path is capped, and the cap is the design: retrieve top-8 by hybrid score within namespace = user:4417, cut to a 400-token budget, inject as a labelled block. Not the whole store. Eight rows.

That cap is the difference between a memory system and a slow-motion context overflow. Same discipline as the step cap and allowlist in an agent loop.

How to tell whether it qualifies

Four tests. All four, not three.

1. There is a write decision. Something chooses what to persist without a human filling in a form. If persistence only happens when a user clicks "save note," that is a notes feature.
2. It survives a new thread and a process restart. Open a fresh session, no shared history, and the fact still shows up.
3. Retrieval is selective and bounded. A fixed top-k and a token budget. "We append the transcript" fails.
4. A single fact can be read and deleted. ChatGPT's February 2024 release made memories viewable and deletable per entry; if you cannot show a user the row, you cannot honour a deletion request against it.

Unbounded growth is the first thing that breaks. The store fills with near-duplicates and retrieval returns eight versions of the same preference. Dedupe on write — dedupe on read means you pay embedding and ranking cost on garbage forever, on every turn.

Then stale facts. A superseded decision keeps winning because it was scored on relevance and never invalidated. Temporal approaches like Graphiti exist because of exactly this.

Memory poisoning is the one that should worry you. Text arrives in a tool result or a scraped page, gets written as a fact, and is re-injected as trusted context on every run after that. Indirect prompt injection with a persistence layer attached. The write path needs its own allowlist.

I would not let an agent write to memory and act on that memory inside the same run without a diff a human can read. Not because the model is stupid. Because a bad row written at 11:20 gets read back on every run after it, and nobody goes looking for it.