I9 · Generative AI and Large Language Models in Practice

Building a Chatbot with Memory

A chatbot with memory keeps track of a conversation and re-sends the relevant parts with every request. Short-term memory is the recent message history, trimmed to fit the context window. Summary memory replaces older turns with a running summary. Long-term memory stores facts or past conversations outside the model, for example in a database or vector store, and retrieves the relevant pieces when needed. Designing memory means choosing what to keep, what to forget and how to protect the user's privacy.

15 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

An LLM API has no memory: each request starts from nothing. Yet chat apps remember what you said ten messages ago, and some even recall your preferences next week. The memory lives in your code, not in the model.

A chatbot with memory keeps track of a conversation and re-sends the relevant parts with every request. Short-term memory is the recent message history, trimmed to fit the context window. Summary memory replaces older turns with a running summary. Long-term memory stores facts or past conversations outside the model, for example in a database or vector store, and retrieves the relevant pieces when needed. Designing memory means choosing what to keep, what to forget and how to protect the user's privacy.

In simple words

Imagine a helpful shopkeeper who cannot remember anything, but keeps a notebook. Before answering, they read the last few lines of your conversation and a short note about you, such as "prefers Hindi, allergic to peanuts". The notebook is the chatbot's memory.

Overview

  • LLM APIs are stateless: the model remembers nothing between calls unless you send it again.
  • A chatbot's "memory" is the conversation history and facts the program chooses to include in each request.
  • Long histories hit the context window and raise cost, so memory must be managed.
  • Common strategies keep recent turns, summarise older ones and store key facts separately.
  • Long-term memory can be stored in a database or vector store and retrieved when relevant.
  • Users should be told what is remembered and be able to delete it.

Why it matters

Memory makes assistants feel coherent and personal: tutors that remember your weak topics, support bots that do not ask for your order number twice. Poorly designed memory wastes tokens, confuses the model or leaks private information.

Key terms

Stateless
Each API request is independent; the model keeps nothing between calls.
Conversation history
The list of previous user and assistant messages re-sent with each request.
Sliding window
Keeping only the most recent messages that fit a token budget.
Summary memory
A running summary of older turns that replaces them.
Long-term memory
Facts or past chats stored outside the model and retrieved when relevant.
Session
One conversation, identified so its history can be stored and resumed.

Types

There are several types of chatbot memory. Real chatbots usually combine more than one.

1. Full conversation buffer

Sending the entire conversation history with every request.

  • How it works: Each user and assistant message is appended to a list that is resent in full, so the model sees everything said so far.
  • Example: A short help-desk chat of ten messages that fits easily in the context window.
  • When to use it: Use it for short conversations; it becomes slow and expensive as chats grow.

2. Sliding-window memory

Keeping only the most recent N messages or tokens.

  • How it works: Older messages are dropped once the limit is reached, so cost stays fixed but early details are forgotten.
  • Example: A tutoring bot keeping the last 20 messages of a long study session.
  • When to use it: Use it when recent context matters most and older details can be lost.

3. Summary memory

Replacing older parts of the conversation with a running summary.

  • How it works: When the history grows long, the model is asked to summarise it; the summary plus recent turns are sent next time.
  • Example: "Summary so far: the user is planning a 3-day trip to Jaipur on a budget of ₹15,000."
  • When to use it: Use it for long conversations where the gist matters more than exact words.

4. Entity and profile memory

Storing key facts about the user or topics in a structured form.

  • How it works: The program extracts facts such as name, preferences and goals into a profile that is added to the system prompt each time.
  • Example: Remembering that a learner prefers Hindi explanations and is studying for an exam in March.
  • When to use it: Use it for personalisation, with clear consent and an option to edit or delete.

5. Retrieval-based long-term memory

Saving past messages as embeddings and retrieving relevant ones when needed.

  • How it works: Each past exchange is embedded and stored in a vector database; a new question retrieves the most similar memories to include.
  • Example: Recalling a recipe the user discussed three weeks ago when they ask about it again.
  • When to use it: Use it for assistants used over months, where full history would never fit.

Explained in detail

Three layers of memory

The simplest chatbot keeps a list of messages and appends each new user message and assistant reply, sending the whole list every time. This works until the conversation outgrows the context window or becomes expensive, because every request re-sends every earlier token. A sliding window keeps only the last N messages or the last N thousand tokens. Summary memory goes further: when the history gets long, the bot asks the model to summarise the older part in a few sentences, keeps that summary at the top and drops the original messages. Long-term memory stores durable facts, such as "student is preparing for the state engineering entrance exam" or "prefers answers in Marathi", in a database keyed to the user, and inserts them into the system prompt at the start of each session. For large histories, past messages can be embedded and the most relevant ones retrieved, like RAG over the user's own conversations.

Memory typeStrengthWeakness
Full historyNothing is lostGrows costly; eventually exceeds the window
Sliding windowSimple, predictable costForgets early details such as the user's name
Summary memoryKeeps the gist cheaplySummaries can drop or distort details
Long-term factsPersonalises across sessionsNeeds consent, editing and deletion controls

Privacy and control

Memory means storing what people say, which may include health details, family matters or financial information. Tell users what is remembered, let them view, edit and delete it, store it securely and keep it only as long as needed. India's Digital Personal Data Protection Act, 2023 gives people rights to information about, correction of and erasure of their personal data, and these apply to chatbot memory too. Never let one user's memory leak into another user's session: key all stored memory to an authenticated user and test this carefully.

Memory also has a cost. Every token of history, summary and stored facts is re-sent with each request, so a bot that keeps 3,000 tokens of memory pays for those tokens on every turn. Good designs keep memory compact: short summaries, a handful of facts and only the most relevant retrieved messages. Summaries should be checked from time to time, because a model can drop an important detail or record something the user never said, and an error in memory repeats in every later answer. Giving users a way to see and correct what the bot believes about them fixes such errors and builds trust.

How it works

  1. Store each session's messages, keyed to the user.
  2. Before each request, build the context: system prompt, long-term facts, summary and recent messages.
  3. Trim to a token budget, summarising older turns when needed.
  4. Save new durable facts only with the user's knowledge, and let them edit or delete memory.
  5. Test that memory never leaks between users.

In code

chat_memory.py
class ChatMemory:
    def __init__(self, budget_words=25):
        self.facts = {}            # long-term memory, kept across sessions
        self.summary = ""          # summary of older turns
        self.recent = []           # short-term memory: recent (role, text) pairs
        self.budget = budget_words

    def add(self, role, text):
        self.recent.append((role, text))
        while sum(len(t.split()) for _, t in self.recent) > self.budget:
            old_role, old_text = self.recent.pop(0)
            self.summary += f" {old_role} said: {' '.join(old_text.split()[:6])}..."   # a crude summariser

    def remember(self, key, value):
        self.facts[key] = value

    def build_context(self, system):
        facts = ", ".join(f"{k}={v}" for k, v in self.facts.items())
        lines = [f"SYSTEM: {system}", f"KNOWN FACTS: {facts or 'none'}"]
        if self.summary:
            lines.append(f"EARLIER (summary):{self.summary}")
        lines += [f"{r.upper()}: {t}" for r, t in self.recent]
        return "\n".join(lines)

mem = ChatMemory()
mem.remember("name", "Meera")
mem.remember("language", "Hindi")
for role, text in [("user", "I am preparing for my board exams in chemistry"),
                   ("assistant", "Great, which chapter feels hardest right now?"),
                   ("user", "Chemical bonding is confusing, especially ionic versus covalent bonds"),
                   ("assistant", "Ionic bonds transfer electrons; covalent bonds share them."),
                   ("user", "Give me one example of each please")]:
    mem.add(role, text)
print(mem.build_context("You are a patient chemistry tutor."))
print("\nWords sent:", len(mem.build_context("You are a patient chemistry tutor.").split()))

Output

SYSTEM: You are a patient chemistry tutor.
KNOWN FACTS: name=Meera, language=Hindi
EARLIER (summary): user said: I am preparing for my board... assistant said: Great, which chapter feels hardest right...
USER: Chemical bonding is confusing, especially ionic versus covalent bonds
ASSISTANT: Ionic bonds transfer electrons; covalent bonds share them.
USER: Give me one example of each please

Words sent: 56

Try it yourself

A tutor that remembers

A study app's tutor bot is used by Meera every evening for a month.

Match the memory type to the example

Which memory is being used?

  • Re-sending the last 10 messages
  • "Earlier: user asked about refunds for order 551"
  • "User prefers Tamil" stored across sessions
  • Searching past chats for the most relevant ones

Common mistakes

  • Assuming the API remembers earlier requests.
  • Re-sending the entire history forever, so cost keeps growing.
  • Storing personal details without telling users or letting them delete them.

Check yourself

Question 1

1. Where does a chatbot's memory actually live?

Question 2

2. What is the main weakness of a sliding window?

Question 3

3. Which practice protects privacy in chatbot memory?

Summary

  • LLM APIs are stateless; memory is built in your application.
  • Combine recent messages, a running summary and long-term facts within a token budget.
  • Make memory transparent, editable, deletable and strictly separated between users.

Finished reading?

Sources: Anthropic API documentation: Messages · Digital Personal Data Protection Act, 2023 (MeitY)