Agent Memory: Five Checks Beyond Retrieval

“We've added memory.” Open the implementation, and it stores chat history in a vector database, then retrieves the top matching passages when a question arrives.

That can be part of a memory system. It doesn't tell you what happens when a user changes their mind, corrects a fact, or asks for something to be forgotten.

A vector database finds material by similarity; top-k retrieval returns a chosen number of the highest-ranked matches. To judge an agent's memory, look beyond that lookup to the state the application keeps and how it manages it. These five checks follow the places that design can break down.

1. Memory Needs More Than a Retrieval Step

Storing conversations and retrieving passages gives the agent access to earlier material. The next question is whether the application manages that material as circumstances change.

Compare a simple, static document-retrieval setup with the behavior a stateful application may need:

• Data sources: a fixed document collection versus conversations and events that keep arriving.
• Writing: one initial import versus decisions about what to save after later interactions.
• Updating: unchanged source documents versus corrections, superseded facts, and removal requests.
• Personalization: one shared collection versus information scoped to the right user, project, or session.
• Time: matching a passage by topic versus distinguishing what was true last week from what is true now.
• Conflicts: returning matching passages versus deciding what to do when those passages disagree.

The static example is one retrieval setup, not the definition of retrieval-augmented generation, or RAG. A RAG application can update its sources, filter by user, and use time metadata. Retrieval can be a component of memory; those capabilities aren't exclusive to one label.

Without the relevant policies, four failures are possible: a correction leaves the old fact in use; contradictory statements reach the model without clarification; accumulated material buries useful information; or a question such as “the one I mentioned last week” gets an answer from the wrong period.

The distinction to inspect is the lifecycle: what is written, updated, removed, and reconciled, as well as what is retrieved.

2. Give Different Kinds of Memory Different Treatment

One useful way to organize the design is to distinguish four kinds of information. This is a classification you can use, not a requirement to build four databases.

• Working memory: the context available for the current task, including material in the model's context window.
• Episodic memory: a record of an event, such as a user previously asking for a hotel booking.
• Semantic memory: a fact or preference extracted from interactions, such as a stated preference for vegetarian food.
• Procedural memory: a way of working, such as the user's preference for a direct answer to a coding question.

Those categories raise different lifecycle questions. A session's working context might be discarded, summarized, or persisted. An event record needs its time and source, but it needn't be permanently append-only. A preference may need correction or a limit on where it applies. A working convention may be reused often and still change when the user asks.

Putting everything into one store without distinguishing its purpose is like piling all the furniture into one room. The problem is the missing organization, not the room count: one store can carry categories, access scopes, timestamps, and update rules.

Implementations organize this differently. Letta's v1 documentation describes editable memory blocks available in context, alongside archival memory queried through semantic search. Mem0 describes extracting facts from messages and retrieving relevant memories with scope filters, with explicit operations for updates and deletions. Neither example makes the four categories above a universal architecture.

3. Decide What Deserves to Be Written

It's tempting to spend all the effort on embeddings, reranking, and hybrid search: how can retrieval find a better match?

The write path needs just as concrete a design. If a correction never reaches storage, or every passing remark becomes a lasting preference, better ranking won't settle what the application should remember.

Four failure modes are worth checking:

• Save everything: the store fills with turns that add little reusable information.
• Save nothing: useful context exists only during the current task and isn't available in the next session.
• Save noise as facts: greetings and acknowledgments compete with decisions and constraints.
• Keep only bulky transcripts: retrieval brings back more text than the model needs, while the relevant fact remains hard to identify.

There are several ways to improve that path. Decide whether an interaction contains information worth reusing. Extract the useful part while retaining its conditions, time, and source. Use structured entities and relationships where they help the application. Treat a stated preference, plan, constraint, or complaint as a candidate for storage, then check its scope and whether the application is permitted to retain it.

Compression needs its own check. Turning a long conversation into one line can drop the exception that made the preference useful. “For this trip, keep it inexpensive” should not silently become a permanent instruction to choose the cheapest option.

What you write limits what later retrieval can recover. A well-matched passage can still contain the wrong fact or a preference that applies to a different task.

Set the boundaries before writing, too. Temporary instructions may belong only to the current task. Sensitive information needs the appropriate access and retention treatment. A momentary error shouldn't become a lasting fact merely because it appeared in the conversation. Save information for a defined future use, within the application's applicable rules.

4. Handle Forgetting, Updates, and Conflicts

Keeping state across sessions makes these questions unavoidable. A retrieval call alone doesn't answer them.

Forgetting: does last week's detail still matter? Has a preference been replaced? When storage or context is limited, what should be retained, summarized, or removed?

Updates: suppose a user says “I like coffee,” then later says “I've stopped drinking it.” The earlier statement can remain part of the history without remaining the current preference. A move to a new home similarly changes the status of the old location.

Conflicts: two statements may disagree because one corrects the other. They may also concern different projects, be hypothetical, or quote someone else. Simply putting both into the next prompt leaves the model to untangle a distinction the application could have preserved.

Four design choices help make that work explicit. Keep timestamps, sources, and any confidence estimates, while recognizing that recency and confidence don't establish truth. Reconcile conflicting entries by checking their meaning and scope. Consolidate fragmented information without losing important qualifications. If useful for the product, decay the ranking of older or rarely used material—but treat an explicit deletion request according to the removal policy, rather than merely lowering its search score.

A store that only grows can keep serving obsolete information. Updating its status is part of keeping it useful.

5. Test More Than Immediate Recall

An agent answering “What did I just tell you?” can demonstrate recall in that moment. It doesn't establish that the same system will handle a correction, another session, or a deletion correctly.

Build checks around at least these six questions:

• Write accuracy: did the system retain the intended information and leave out material it shouldn't store?
• Retrieval accuracy: did it find the relevant records for the current question and scope?
• Temporal consistency: after a fact changed, did the agent use the appropriate version?
• Conflict handling: when you supplied contradictory statements, did the system preserve their context and resolve them appropriately?
• Long-term consistency: after an extended conversation—for example, fifty turns—did the agent's account of the user's preferences drift?
• Forgetting: after a removal request, could the application still retrieve or use the information through a store, index, summary, or other accessible path?

For that last check, define the deletion and retention boundary you are testing. One answer that happens not to mention a fact doesn't demonstrate its removal.

LoCoMo and LongMemEval provide benchmark material for evaluating long-term conversational memory. They can be starting points; add cases that reflect your application's actual tasks. A benchmark's existence, or another system's reported score, doesn't establish that your write, update, or deletion behavior works.

Without those checks, a convincing demonstration can leave cross-session behavior largely unexamined.

6. Finish with Four Questions

Reasoning and tool use are visible parts of an agent demo. State management deserves a similarly concrete explanation: what will the agent carry into its next task, and why?

When assessing a memory implementation, ask:

1. What kinds of information does it distinguish, and how are their lifecycles and access scopes handled?
2. What is the write policy, and how does extraction or compression preserve important conditions?
3. How does it handle forgetting, updates, and conflicting information?
4. How are those behaviors measured on representative cases?

Clear answers give you design decisions to inspect and behavior to test. That's a stronger basis for judging memory than the presence of a vector database—or the number of layers drawn on a slide.