Give your chatbot a memory: session state that actually works

Every LLM API is stateless. Each request arrives as a blank slate with no recollection of what was discussed five minutes ago, and any coherence your chatbot shows comes from you feeding history back into the prompt. The naive approach — appending the whole transcript every turn — works until the conversation gets long, the bill gets large, and the model starts losing track of what matters.

This tutorial covers the three memory architectures that survive long conversations: summarization, retrieval, and entity stores. For each you'll see how it works, when it fits, and where it breaks. At the end, a decision table helps you pick.

The problem: a context window is not memory#

Stuffing the full transcript into every prompt fails for three practical reasons:

1. Cost and latency grow with the conversation. Every token is re-processed on every turn. A long support session or multi-hour coding task accumulates hundreds of thousands of tokens.
2. Attention degrades over distance. Real conversations are thematically discontinuous — a user jumps from billing to a feature request and back. Distant tokens get diluted even when they fit in the window.
3. Sessions reset. The window covers one session. Anything the user expects you to know tomorrow needs to live outside the prompt entirely.

Frameworks like LangChain codified the classic workarounds: keep a verbatim transcript for short high-precision tasks, keep a sliding window of the last few turns for predictable interactions, and reach for heavier patterns beyond that. The three architectures below are those heavier patterns.

Architecture 1: Summarization#

Instead of carrying every word, you maintain a rolling summary of the conversation. After each exchange (or every N exchanges), a model call condenses the important points — decisions made, facts stated, current goal — into a short paragraph. Only that summary, plus the most recent turns verbatim, goes into the next prompt.

Why it works: context size stays roughly flat no matter how long the session runs.

Where it breaks: summarization is lossy by design. Specifics — an exact error code, a quoted number — get smoothed away, and errors from a cheaper summarizer model compound over turns.

Making it robust:

• Keep a verbatim window of the last few turns alongside the summary, so recent specifics aren't lost.
• Write summaries structured, not freeform: current goal, open questions, decisions, key facts. Structure resists drift.
• Re-summarize from source material when the summary gets long, rather than summarizing summaries forever.

This has gone mainstream enough that platforms now productize it: Anthropic shipped automatic context compaction with Claude Opus 4.6 (2026), summarizing older conversation parts as a threshold approaches instead of every developer hand-rolling truncation logic. Convenient — but the keep-or-compress judgment moves into the model's hands, which isn't always right. For anything load-bearing, keep your own summarization layer on top.

Architecture 2: Retrieval-based memory#

Retrieval treats conversation history as a database. Each exchange (or each extracted fact) is embedded and stored in a vector store. At inference time, you embed the current user message, run a similarity search, and inject only the top relevant chunks into the prompt. History stops growing the window; the window stays a fixed size no matter how many sessions pile up.

The best-known implementation of this pattern is Mem0, an open-source memory layer built around a three-phase pipeline (described in its paper, Chhikara et al., arXiv 2504.19413):

1. Extraction. Given a new message pair plus a rolling conversation summary, an LLM extraction function pulls out candidate salient facts — preferences, decisions, goals.
2. Update. Each candidate fact is checked against the most semantically similar existing memories. The LLM then decides the operation directly: ADD (new fact), UPDATE (augment an existing memory), DELETE (contradicted), or no-op. There is no separate classifier — the model reasons about the semantic relationship through tool calls.
3. Retrieval. At query time, the query is embedded and the top relevant memory facts are injected into the prompt — instead of re-processing the whole transcript.

The tradeoff is that extraction is a guess — the LLM decides at write time what will matter at read time.

In practice it looks like this (from Mem0's official docs):

from mem0 import Memory

memory = Memory()

memory.add(
    "I'm Alex and I prefer boutique hotels.",
    user_id="alex",
    run_id="trip-planning-2025",
)

results = memory.search(
    "Any hotel preferences?",
    filters={"user_id": "alex", "run_id": "trip-planning-2025"},
)

Scope matters here: use a session- or task-scoped ID for things tied to one conversation, and a bare user_id for anything that should persist across every session for that person.

Where retrieval breaks: embedding search is similarity, not understanding. It retrieves what sounds related, which fails for negation ("I do not want..."), time-dependent facts ("my budget was $500, now it's $800"), and anything the user never phrased in retrievable words. Retrieval also has a cold-start problem: early in a relationship, there's nothing to retrieve.

Architecture 3: Entity stores#

Some applications don't need the dialogue at all — they need facts about things. An entity store extracts named participants and topics (people, projects, products, preferences) and maintains a structured record per entity. When the topic resurfaces, relevant facts are pulled from storage.

Example: a personalized coding coach. The user mentions they prefer React and deploy on a particular cloud provider. Those facts get stored as entities. Weeks later, when they ask for a code sample, the bot applies those preferences without rereading the original transcript.

Mem0's graph variant (sometimes called Mem0g) takes this further: memories become a directed labeled graph where nodes are entities and edges are relationships (user → prefers → React). Retrieval combines entity-centric graph traversal with semantic search, which enables multi-hop reasoning.

Where entity stores break: extraction errors are structural. A misidentified entity or two people merged under one name corrupt the store silently. Entity stores also need conflict handling by design — the user who "prefers React" in January and "prefers Svelte" in June needs an UPDATE, not two contradictory facts.

The piece most tutorials skip: memory is a lifecycle#

A useful framing, popularized in recent writing on agent memory, is that every memory must go through three stages: formation (what gets written), evolution (how it's updated, merged, or forgotten), and retrieval (how it's found and used). Most systems implement only retrieval. Without the other two, long-term memory becomes bloated, contradictory, and outdated — which is why so many chatbots feel coherent in week one and subtly wrong by week four.

Practically, this means budgeting for governance, not just storage: salience detection at write time (drop trivia), explicit conflict resolution (update or delete contradictions, never let both coexist), aging and decay (weight recent facts more, or re-confirm old ones), and periodic consolidation of overlapping memories.

This is the layer most hobby projects skip and most production systems can't live without. Memory is not a database problem; it's a governance problem.

Choosing the right architecture#

The patterns aren't mutually exclusive — production systems layer them. But start with the one that matches your bottleneck:

A sensible default for a chatbot that should "remember" across sessions: a short verbatim window (recent turns), a rolling summary (session narrative), a retrieval layer over extracted facts (long-term memory scoped by user), and an entity profile (durable preferences). Write to all of them each turn; read from all of them each turn; and give every write path an explicit update-or-delete decision.

The takeaway#

Long conversations don't fail because the context window is too small — they fail because developers treat the window as memory. The window is a workspace; memory is what you build around it: summaries that keep the narrative flat, retrieval that pulls in only what's relevant, and entity stores that track the facts worth keeping. The differentiator between a demo and a system that holds up over weeks of use isn't which vector database you pick. It's whether you manage the full lifecycle — formation, evolution, retrieval — or just the last step and hope for the best.

One more caution, from Mem0's own documentation: avoid storing secrets or unredacted personal data in memories. They are retrievable by design — that's the point — so encrypt or hash sensitive values before they ever reach the memory layer.