AI Agent Memory Systems — Short-Term, Long-Term, and Vector vs File-Based

"Agent memory" gets used loosely to mean several different things — the current context window, a running summary, and persistent facts across sessions are all different problems with different correct solutions. Conflating them is what causes agents that either forget things they should remember or waste tokens re-loading things they don't need.

Working Memory: The Context Window Itself

Working memory is just whatever is currently in the model's context — inherently short-lived and hard-bounded by the context window size.

The important design decision isn't how to build working memory — it's what happens when it fills up. An agent with no overflow strategy either crashes or silently truncates from the front, losing whatever was there first (often the original task description).

Session Memory: A Running Summary of the Current Task

Session memory bridges the gap when a task spans more turns than comfortably fit in context — instead of re-sending the full turn-by-turn history, older turns get compacted into a summary.

Using a smaller, cheaper model for the summarization call itself keeps this compaction step from adding meaningful cost, since it happens repeatedly over a long session.

Long-Term Memory: Storage Is the Easy Part, Retrieval Is the Hard Part

Persisting data across sessions is trivial — write it to a database. The actual design problem is how the agent finds the right piece of stored information later, out of everything it's ever stored.

Vector-Based Retrieval for Fuzzy, Semantic Recall

This is the right tool for "what have we discussed related to this topic" — fuzzy, semantic recall where the exact wording of the original information isn't known in advance.

File-Based or Structured Retrieval for Exact Facts

For facts that must be retrieved reliably every time — a user's subscription tier, a project's configured settings — a direct key lookup is simpler, faster, and doesn't carry the failure mode of semantic search silently missing the right result because it was phrased unusually.

Why Relying on Vector Search Alone Causes Silent Failures

Semantic similarity search returns the most similar items to a query, not necessarily the correct or complete set. A structurally important fact that happens to be phrased in dissimilar language to the current query can simply not appear in the top results — with no error, no exception, just an agent that behaves as if it never learned that fact.

Combining both — guaranteed retrieval for facts that must always surface, best-effort semantic search for everything else — avoids the failure mode of either approach used alone.

Choosing a Design for a Given Agent

A customer support agent that needs to recall "what did we discuss with this user before" benefits from vector-based memory. The same agent needing to know "is this user on the Pro plan" should never rely on semantic search for that — it's a direct lookup. Most production agents need both layers, matched to the shape of what they're trying to remember.

Key Takeaways

Working memory, session memory, and long-term memory are three different problems — bounding context, compacting a running session, and persisting facts across sessions — that need different solutions, and long-term memory specifically needs both guaranteed structured lookup for facts that must always surface and vector-based semantic search for fuzzy recall, since relying on vector search alone causes silent, hard-to-detect recall failures.