I9 · Generative AI and Large Language Models in Practice

Tokens and Context Windows

A large language model (LLM) reads and writes tokens, small pieces of text produced by a sub-word tokeniser. Its context window is the maximum number of tokens it can consider at once, counting the system instructions, the whole conversation so far, any pasted documents and the reply it is writing. Anything outside the window is invisible to the model. Context windows have grown from about 2,000 tokens in early GPT-3 models (2020) to hundreds of thousands or more in many models today, but cost and speed still grow with the number of tokens.

8 min Intermediate · Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

Paste a long document into a chatbot and it may say the text is too long, or quietly forget the beginning of your conversation. Both happen because a language model sees text as tokens and can only look at a limited number of them at once.

A large language model (LLM) reads and writes tokens, small pieces of text produced by a sub-word tokeniser. Its context window is the maximum number of tokens it can consider at once, counting the system instructions, the whole conversation so far, any pasted documents and the reply it is writing. Anything outside the window is invisible to the model. Context windows have grown from about 2,000 tokens in early GPT-3 models (2020) to hundreds of thousands or more in many models today, but cost and speed still grow with the number of tokens.

In simple words

Imagine reading through a window that shows only one page at a time. To look at a new page, an old page must slide out of view. A model's context window works the same way, except the pages are tokens.

Overview

  • LLMs read and write tokens: pieces of words, whole words or punctuation.
  • In English, one token averages about four characters or three-quarters of a word.
  • Indian-language text often needs more tokens per word, which raises cost.
  • The context window is the maximum number of tokens the model can consider at once, including the prompt and the answer.
  • Context windows have grown from 2,048 tokens (GPT-3, 2020) to hundreds of thousands or more.
  • APIs charge per input and output token, so token counts drive cost and speed.

Why it matters

Tokens decide what an LLM can see, how much it costs and how fast it answers. Knowing them helps you write prompts that fit, design chatbots that remember the right things and estimate the bill before you build.

Key terms

Token
A piece of text, often part of a word, that the model processes as one unit.
Context window
The maximum number of tokens of input plus output the model can handle at once.
Input and output tokens
Tokens you send and tokens the model generates; they are often priced differently.
Truncation
Cutting text that does not fit, usually the oldest messages.
Max output tokens
A limit you set on how long the reply may be.
Lost in the middle
The tendency of models to use information at the start and end of a long context better than the middle.

Types

Tokens and context limits come in several types. Understanding each helps you design prompts and applications.

1. Text tokens (subwords)

Units produced by a subword tokeniser such as byte-pair encoding.

  • How it works: Common words become one token; rare words split into several; spaces and punctuation are usually part of tokens.
  • Example: "tokenisation" might split into "token", "isation".
  • When to use it: Use a tokeniser tool to count tokens before sending long prompts, since limits and prices are set in tokens.

2. Special tokens

Reserved tokens that mark structure, such as the start of text, the end of text and roles in a chat.

  • How it works: Chat templates wrap system, user and assistant messages in special tokens so the model knows who is speaking.
  • Example: An end-of-sequence token telling the model to stop generating.
  • When to use it: Recognise them when working with open models, where the correct chat template must be used.

3. Image, audio and other tokens

Non-text inputs converted into token-like vectors.

  • How it works: Images are split into patches and encoded; each image can cost hundreds of tokens depending on its size.
  • Example: Uploading a chart to a multimodal model, which counts it as many tokens of context.
  • When to use it: Use smaller or cropped images when only part of the picture matters, to save context and cost.

4. Context window

The total token budget for instructions, history, documents and the reply.

  • How it works: Anything beyond the window is cut off or must be summarised; models can also attend less carefully to material in the middle of very long contexts.
  • Example: A 200-page report that does not fit, so only relevant sections are retrieved and included.
  • When to use it: Plan what goes into the window: put key instructions first or last, and retrieve rather than paste everything.

5. Output limits and rate limits

Caps on how many tokens a reply can contain and how many tokens or requests are allowed per minute.

  • How it works: max_tokens limits the reply length; providers also limit tokens per minute, so large jobs must be batched and retried.
  • Example: A summary cut off mid-sentence because max_tokens was set too low.
  • When to use it: Set output limits deliberately and handle rate-limit errors with retries and back-off.

Explained in detail

From text to tokens

Before an LLM sees your message, a tokeniser splits it into tokens from a fixed vocabulary, often 50,000 to 200,000 entries, and replaces each by an integer ID. Common English words are usually one token each; longer or rarer words split into pieces; spaces and punctuation are part of tokens too. As a rough rule, 1,000 English tokens are about 750 words. Hindi, Tamil and other Indian languages often need more tokens for the same meaning, especially with tokenisers trained mostly on English, so the same message can cost more and fill the window faster. Numbers are tokenised in odd ways, which is one reason models sometimes stumble on arithmetic with long numbers.

What fills the context window

PartExampleTip
System instructions"You are a helpful tutor for Indian students..."Keep them clear and short; they are sent with every request
Conversation historyAll earlier user and assistant messagesTrim or summarise old turns
Documents and tool resultsA pasted PDF, search results, database rowsInclude only the relevant parts
The replyThe tokens the model is generatingReserve space with a max-output setting

Why limits exist and how to work with them

In a transformer, every token attends to every other token, so the work of attention grows roughly with the square of the context length, and the memory needed to store intermediate values grows too. Engineering advances have pushed windows much larger, but long contexts still cost more money and time, and research has found that models often use information in the middle of very long inputs less reliably than information near the start or end, an effect described in a 2023 paper titled "Lost in the Middle". In practice, put the most important instructions at the start, put the question near the end, remove irrelevant text, and for large document collections use retrieval to select only the passages that matter rather than pasting everything.

  • Count tokens with the provider's tokeniser before sending large inputs.
  • Set a maximum output length so replies cannot run on and cost more than expected.

Many providers also offer prompt caching, which charges less for a long instruction or document that is sent unchanged with many requests.

How it works

  1. Estimate tokens for your instructions, typical messages and documents.
  2. Check the model's context window and price per million input and output tokens.
  3. Reserve room for the reply with a max-output setting.
  4. Trim, summarise or retrieve so only relevant text is sent.
  5. Put key instructions first and the question last.

In code

context_window.py
import re

def rough_tokens(text):
    # A crude stand-in for a real tokeniser: words, punctuation, and long words split every 6 letters
    pieces = []
    for w in re.findall(r"\w+|[^\w\s]", text):
        pieces += [w[i:i + 6] for i in range(0, len(w), 6)]
    return pieces

print(rough_tokens("Tokenisation splits unbelievably long words!"))

system = "You are a friendly tutor."
history = [("user", "Hi, my name is Asha and I live in Indore."), ("assistant", "Hello Asha! How can I help?"),
           ("user", "Explain photosynthesis simply."), ("assistant", "Plants use sunlight, water and CO2 to make food and oxygen."),
           ("user", "Now explain respiration in plants."), ("assistant", "Plants break down food to release energy, using oxygen."),
           ("user", "What is my name?")]
WINDOW, RESERVED_FOR_REPLY = 60, 15

def fit(system, history):
    budget = WINDOW - RESERVED_FOR_REPLY - len(rough_tokens(system))
    kept = []
    for role, text in reversed(history):          # keep the most recent turns first
        cost = len(rough_tokens(text))
        if cost > budget:
            break
        kept.insert(0, (role, text))
        budget -= cost
    return kept

kept = fit(system, history)
print(f"Kept {len(kept)} of {len(history)} messages")
print("Is the name still visible?", any("Asha" in t for _, t in kept))
price_in, price_out = 3.0, 15.0                      # example prices per million tokens
print(f"Cost of 1,000 chats of 2,000 in + 500 out tokens: ${1000 * (2000 * price_in + 500 * price_out) / 1e6:.2f}")

Output

['Tokeni', 'sation', 'splits', 'unbeli', 'evably', 'long', 'words', '!']
Kept 3 of 7 messages
Is the name still visible? False
Cost of 1,000 chats of 2,000 in + 500 out tokens: $13.50

Try it yourself

The chatbot that forgot

A coaching centre's study bot chats with a student for an hour.

Inside the context or not?

Which of these count towards the context window of one request?

  • The reply being generated

  • The model's training data

  • The system instructions

  • Earlier messages included in this request

  • A conversation from last week that was not sent

Common mistakes

  • Assuming the model remembers earlier chats that were not sent again.
  • Pasting whole documents when only a few passages matter.
  • Forgetting that Indian-language text may use many more tokens.

Check yourself

Question 1

1. Roughly how many English words are 1,000 tokens?

Question 2

2. What happens to text outside the context window?

Question 3

3. Where should the key question usually go in a long prompt?

Summary

  • LLMs read and write tokens; 1,000 English tokens are about 750 words.
  • The context window holds instructions, history, documents and the reply; anything outside is invisible.
  • Trim, summarise and retrieve to stay within limits and control cost.

Finished reading?

Sources: Hugging Face course: tokenizers and transformer models · Jurafsky and Martin, Speech and Language Processing (3rd ed. draft): Large Language Models