Definition
An LLM API has no memory: each request starts from nothing. Yet chat apps remember what you said ten messages ago, and some even recall your preferences next week. The memory lives in your code, not in the model.
A chatbot with memory keeps track of a conversation and re-sends the relevant parts with every request. Short-term memory is the recent message history, trimmed to fit the context window. Summary memory replaces older turns with a running summary. Long-term memory stores facts or past conversations outside the model, for example in a database or vector store, and retrieves the relevant pieces when needed. Designing memory means choosing what to keep, what to forget and how to protect the user's privacy.
In simple words
Imagine a helpful shopkeeper who cannot remember anything, but keeps a notebook. Before answering, they read the last few lines of your conversation and a short note about you, such as "prefers Hindi, allergic to peanuts". The notebook is the chatbot's memory.
Overview
- LLM APIs are stateless: the model remembers nothing between calls unless you send it again.
- A chatbot's "memory" is the conversation history and facts the program chooses to include in each request.
- Long histories hit the context window and raise cost, so memory must be managed.
- Common strategies keep recent turns, summarise older ones and store key facts separately.
- Long-term memory can be stored in a database or vector store and retrieved when relevant.
- Users should be told what is remembered and be able to delete it.
Why it matters
Memory makes assistants feel coherent and personal: tutors that remember your weak topics, support bots that do not ask for your order number twice. Poorly designed memory wastes tokens, confuses the model or leaks private information.
Key terms
- Stateless
- Each API request is independent; the model keeps nothing between calls.
- Conversation history
- The list of previous user and assistant messages re-sent with each request.
- Sliding window
- Keeping only the most recent messages that fit a token budget.
- Summary memory
- A running summary of older turns that replaces them.
- Long-term memory
- Facts or past chats stored outside the model and retrieved when relevant.
- Session
- One conversation, identified so its history can be stored and resumed.
Types
There are several types of chatbot memory. Real chatbots usually combine more than one.
1. Full conversation buffer
Sending the entire conversation history with every request.
- How it works: Each user and assistant message is appended to a list that is resent in full, so the model sees everything said so far.
- Example: A short help-desk chat of ten messages that fits easily in the context window.
- When to use it: Use it for short conversations; it becomes slow and expensive as chats grow.
2. Sliding-window memory
Keeping only the most recent N messages or tokens.
- How it works: Older messages are dropped once the limit is reached, so cost stays fixed but early details are forgotten.
- Example: A tutoring bot keeping the last 20 messages of a long study session.
- When to use it: Use it when recent context matters most and older details can be lost.
3. Summary memory
Replacing older parts of the conversation with a running summary.
- How it works: When the history grows long, the model is asked to summarise it; the summary plus recent turns are sent next time.
- Example: "Summary so far: the user is planning a 3-day trip to Jaipur on a budget of ₹15,000."
- When to use it: Use it for long conversations where the gist matters more than exact words.
4. Entity and profile memory
Storing key facts about the user or topics in a structured form.
- How it works: The program extracts facts such as name, preferences and goals into a profile that is added to the system prompt each time.
- Example: Remembering that a learner prefers Hindi explanations and is studying for an exam in March.
- When to use it: Use it for personalisation, with clear consent and an option to edit or delete.
5. Retrieval-based long-term memory
Saving past messages as embeddings and retrieving relevant ones when needed.
- How it works: Each past exchange is embedded and stored in a vector database; a new question retrieves the most similar memories to include.
- Example: Recalling a recipe the user discussed three weeks ago when they ask about it again.
- When to use it: Use it for assistants used over months, where full history would never fit.
Explained in detail
Three layers of memory
The simplest chatbot keeps a list of messages and appends each new user message and assistant reply, sending the whole list every time. This works until the conversation outgrows the context window or becomes expensive, because every request re-sends every earlier token. A sliding window keeps only the last N messages or the last N thousand tokens. Summary memory goes further: when the history gets long, the bot asks the model to summarise the older part in a few sentences, keeps that summary at the top and drops the original messages. Long-term memory stores durable facts, such as "student is preparing for the state engineering entrance exam" or "prefers answers in Marathi", in a database keyed to the user, and inserts them into the system prompt at the start of each session. For large histories, past messages can be embedded and the most relevant ones retrieved, like RAG over the user's own conversations.
| Memory type | Strength | Weakness |
|---|---|---|
| Full history | Nothing is lost | Grows costly; eventually exceeds the window |
| Sliding window | Simple, predictable cost | Forgets early details such as the user's name |
| Summary memory | Keeps the gist cheaply | Summaries can drop or distort details |
| Long-term facts | Personalises across sessions | Needs consent, editing and deletion controls |
Privacy and control
Memory means storing what people say, which may include health details, family matters or financial information. Tell users what is remembered, let them view, edit and delete it, store it securely and keep it only as long as needed. India's Digital Personal Data Protection Act, 2023 gives people rights to information about, correction of and erasure of their personal data, and these apply to chatbot memory too. Never let one user's memory leak into another user's session: key all stored memory to an authenticated user and test this carefully.
Memory also has a cost. Every token of history, summary and stored facts is re-sent with each request, so a bot that keeps 3,000 tokens of memory pays for those tokens on every turn. Good designs keep memory compact: short summaries, a handful of facts and only the most relevant retrieved messages. Summaries should be checked from time to time, because a model can drop an important detail or record something the user never said, and an error in memory repeats in every later answer. Giving users a way to see and correct what the bot believes about them fixes such errors and builds trust.
How it works
- Store each session's messages, keyed to the user.
- Before each request, build the context: system prompt, long-term facts, summary and recent messages.
- Trim to a token budget, summarising older turns when needed.
- Save new durable facts only with the user's knowledge, and let them edit or delete memory.
- Test that memory never leaks between users.
In code
class ChatMemory:
def __init__(self, budget_words=25):
self.facts = {} # long-term memory, kept across sessions
self.summary = "" # summary of older turns
self.recent = [] # short-term memory: recent (role, text) pairs
self.budget = budget_words
def add(self, role, text):
self.recent.append((role, text))
while sum(len(t.split()) for _, t in self.recent) > self.budget:
old_role, old_text = self.recent.pop(0)
self.summary += f" {old_role} said: {' '.join(old_text.split()[:6])}..." # a crude summariser
def remember(self, key, value):
self.facts[key] = value
def build_context(self, system):
facts = ", ".join(f"{k}={v}" for k, v in self.facts.items())
lines = [f"SYSTEM: {system}", f"KNOWN FACTS: {facts or 'none'}"]
if self.summary:
lines.append(f"EARLIER (summary):{self.summary}")
lines += [f"{r.upper()}: {t}" for r, t in self.recent]
return "\n".join(lines)
mem = ChatMemory()
mem.remember("name", "Meera")
mem.remember("language", "Hindi")
for role, text in [("user", "I am preparing for my board exams in chemistry"),
("assistant", "Great, which chapter feels hardest right now?"),
("user", "Chemical bonding is confusing, especially ionic versus covalent bonds"),
("assistant", "Ionic bonds transfer electrons; covalent bonds share them."),
("user", "Give me one example of each please")]:
mem.add(role, text)
print(mem.build_context("You are a patient chemistry tutor."))
print("\nWords sent:", len(mem.build_context("You are a patient chemistry tutor.").split()))Output
SYSTEM: You are a patient chemistry tutor. KNOWN FACTS: name=Meera, language=Hindi EARLIER (summary): user said: I am preparing for my board... assistant said: Great, which chapter feels hardest right... USER: Chemical bonding is confusing, especially ionic versus covalent bonds ASSISTANT: Ionic bonds transfer electrons; covalent bonds share them. USER: Give me one example of each please Words sent: 56
Try it yourself
A tutor that remembers
A study app's tutor bot is used by Meera every evening for a month.
Match the memory type to the example
Which memory is being used?
- Re-sending the last 10 messages
- "Earlier: user asked about refunds for order 551"
- "User prefers Tamil" stored across sessions
- Searching past chats for the most relevant ones
Common mistakes
- Assuming the API remembers earlier requests.
- Re-sending the entire history forever, so cost keeps growing.
- Storing personal details without telling users or letting them delete them.
Check yourself
Summary
- LLM APIs are stateless; memory is built in your application.
- Combine recent messages, a running summary and long-term facts within a token budget.
- Make memory transparent, editable, deletable and strictly separated between users.
Finished reading?
Sources: Anthropic API documentation: Messages · Digital Personal Data Protection Act, 2023 (MeitY)