Definition
Paste a long document into a chatbot and it may say the text is too long, or quietly forget the beginning of your conversation. Both happen because a language model sees text as tokens and can only look at a limited number of them at once.
A large language model (LLM) reads and writes tokens, small pieces of text produced by a sub-word tokeniser. Its context window is the maximum number of tokens it can consider at once, counting the system instructions, the whole conversation so far, any pasted documents and the reply it is writing. Anything outside the window is invisible to the model. Context windows have grown from about 2,000 tokens in early GPT-3 models (2020) to hundreds of thousands or more in many models today, but cost and speed still grow with the number of tokens.
In simple words
Imagine reading through a window that shows only one page at a time. To look at a new page, an old page must slide out of view. A model's context window works the same way, except the pages are tokens.
Overview
- LLMs read and write tokens: pieces of words, whole words or punctuation.
- In English, one token averages about four characters or three-quarters of a word.
- Indian-language text often needs more tokens per word, which raises cost.
- The context window is the maximum number of tokens the model can consider at once, including the prompt and the answer.
- Context windows have grown from 2,048 tokens (GPT-3, 2020) to hundreds of thousands or more.
- APIs charge per input and output token, so token counts drive cost and speed.
Why it matters
Tokens decide what an LLM can see, how much it costs and how fast it answers. Knowing them helps you write prompts that fit, design chatbots that remember the right things and estimate the bill before you build.
Key terms
- Token
- A piece of text, often part of a word, that the model processes as one unit.
- Context window
- The maximum number of tokens of input plus output the model can handle at once.
- Input and output tokens
- Tokens you send and tokens the model generates; they are often priced differently.
- Truncation
- Cutting text that does not fit, usually the oldest messages.
- Max output tokens
- A limit you set on how long the reply may be.
- Lost in the middle
- The tendency of models to use information at the start and end of a long context better than the middle.
Types
Tokens and context limits come in several types. Understanding each helps you design prompts and applications.
1. Text tokens (subwords)
Units produced by a subword tokeniser such as byte-pair encoding.
- How it works: Common words become one token; rare words split into several; spaces and punctuation are usually part of tokens.
- Example: "tokenisation" might split into "token", "isation".
- When to use it: Use a tokeniser tool to count tokens before sending long prompts, since limits and prices are set in tokens.
2. Special tokens
Reserved tokens that mark structure, such as the start of text, the end of text and roles in a chat.
- How it works: Chat templates wrap system, user and assistant messages in special tokens so the model knows who is speaking.
- Example: An end-of-sequence token telling the model to stop generating.
- When to use it: Recognise them when working with open models, where the correct chat template must be used.
3. Image, audio and other tokens
Non-text inputs converted into token-like vectors.
- How it works: Images are split into patches and encoded; each image can cost hundreds of tokens depending on its size.
- Example: Uploading a chart to a multimodal model, which counts it as many tokens of context.
- When to use it: Use smaller or cropped images when only part of the picture matters, to save context and cost.
4. Context window
The total token budget for instructions, history, documents and the reply.
- How it works: Anything beyond the window is cut off or must be summarised; models can also attend less carefully to material in the middle of very long contexts.
- Example: A 200-page report that does not fit, so only relevant sections are retrieved and included.
- When to use it: Plan what goes into the window: put key instructions first or last, and retrieve rather than paste everything.
5. Output limits and rate limits
Caps on how many tokens a reply can contain and how many tokens or requests are allowed per minute.
- How it works: max_tokens limits the reply length; providers also limit tokens per minute, so large jobs must be batched and retried.
- Example: A summary cut off mid-sentence because max_tokens was set too low.
- When to use it: Set output limits deliberately and handle rate-limit errors with retries and back-off.
Explained in detail
From text to tokens
Before an LLM sees your message, a tokeniser splits it into tokens from a fixed vocabulary, often 50,000 to 200,000 entries, and replaces each by an integer ID. Common English words are usually one token each; longer or rarer words split into pieces; spaces and punctuation are part of tokens too. As a rough rule, 1,000 English tokens are about 750 words. Hindi, Tamil and other Indian languages often need more tokens for the same meaning, especially with tokenisers trained mostly on English, so the same message can cost more and fill the window faster. Numbers are tokenised in odd ways, which is one reason models sometimes stumble on arithmetic with long numbers.
What fills the context window
| Part | Example | Tip |
|---|---|---|
| System instructions | "You are a helpful tutor for Indian students..." | Keep them clear and short; they are sent with every request |
| Conversation history | All earlier user and assistant messages | Trim or summarise old turns |
| Documents and tool results | A pasted PDF, search results, database rows | Include only the relevant parts |
| The reply | The tokens the model is generating | Reserve space with a max-output setting |
Why limits exist and how to work with them
In a transformer, every token attends to every other token, so the work of attention grows roughly with the square of the context length, and the memory needed to store intermediate values grows too. Engineering advances have pushed windows much larger, but long contexts still cost more money and time, and research has found that models often use information in the middle of very long inputs less reliably than information near the start or end, an effect described in a 2023 paper titled "Lost in the Middle". In practice, put the most important instructions at the start, put the question near the end, remove irrelevant text, and for large document collections use retrieval to select only the passages that matter rather than pasting everything.
- Count tokens with the provider's tokeniser before sending large inputs.
- Set a maximum output length so replies cannot run on and cost more than expected.
Many providers also offer prompt caching, which charges less for a long instruction or document that is sent unchanged with many requests.
How it works
- Estimate tokens for your instructions, typical messages and documents.
- Check the model's context window and price per million input and output tokens.
- Reserve room for the reply with a max-output setting.
- Trim, summarise or retrieve so only relevant text is sent.
- Put key instructions first and the question last.
In code
import re
def rough_tokens(text):
# A crude stand-in for a real tokeniser: words, punctuation, and long words split every 6 letters
pieces = []
for w in re.findall(r"\w+|[^\w\s]", text):
pieces += [w[i:i + 6] for i in range(0, len(w), 6)]
return pieces
print(rough_tokens("Tokenisation splits unbelievably long words!"))
system = "You are a friendly tutor."
history = [("user", "Hi, my name is Asha and I live in Indore."), ("assistant", "Hello Asha! How can I help?"),
("user", "Explain photosynthesis simply."), ("assistant", "Plants use sunlight, water and CO2 to make food and oxygen."),
("user", "Now explain respiration in plants."), ("assistant", "Plants break down food to release energy, using oxygen."),
("user", "What is my name?")]
WINDOW, RESERVED_FOR_REPLY = 60, 15
def fit(system, history):
budget = WINDOW - RESERVED_FOR_REPLY - len(rough_tokens(system))
kept = []
for role, text in reversed(history): # keep the most recent turns first
cost = len(rough_tokens(text))
if cost > budget:
break
kept.insert(0, (role, text))
budget -= cost
return kept
kept = fit(system, history)
print(f"Kept {len(kept)} of {len(history)} messages")
print("Is the name still visible?", any("Asha" in t for _, t in kept))
price_in, price_out = 3.0, 15.0 # example prices per million tokens
print(f"Cost of 1,000 chats of 2,000 in + 500 out tokens: ${1000 * (2000 * price_in + 500 * price_out) / 1e6:.2f}")Output
['Tokeni', 'sation', 'splits', 'unbeli', 'evably', 'long', 'words', '!'] Kept 3 of 7 messages Is the name still visible? False Cost of 1,000 chats of 2,000 in + 500 out tokens: $13.50
Try it yourself
The chatbot that forgot
A coaching centre's study bot chats with a student for an hour.
Inside the context or not?
Which of these count towards the context window of one request?
The reply being generated
The model's training data
The system instructions
Earlier messages included in this request
A conversation from last week that was not sent
Common mistakes
- Assuming the model remembers earlier chats that were not sent again.
- Pasting whole documents when only a few passages matter.
- Forgetting that Indian-language text may use many more tokens.
Check yourself
Summary
- LLMs read and write tokens; 1,000 English tokens are about 750 words.
- The context window holds instructions, history, documents and the reply; anything outside is invisible.
- Trim, summarise and retrieve to stay within limits and control cost.
Finished reading?
Sources: Hugging Face course: tokenizers and transformer models · Jurafsky and Martin, Speech and Language Processing (3rd ed. draft): Large Language Models