B4 · Domains of AI: Vision, Language, Speech and Robotics

How Computers Break Down Language

Before a computer can work with text, it splits the text into small pieces called tokens, which may be words, parts of words or punctuation marks, and replaces each token with a number. This process is called tokenisation, and every language model begins with it.

8 min Beginner Jump to the activity

What you need: Nothing. This page starts from the beginning, with no programming and no formulas.

Definition

A computer only understands numbers. So how does it read the word "mango"?

Before a computer can work with text, it splits the text into small pieces called tokens, which may be words, parts of words or punctuation marks, and replaces each token with a number. This process is called tokenisation, and every language model begins with it.

In simple words

Think of building blocks. A sentence is a finished model. To pack it into a box, you break it into separate blocks and note the shape number of each block. The computer does the same with a sentence: break it into tokens, then store the number of each token.

Why it matters

Tokens explain several odd things about language models: why they have a limit on how much text they can read at once, why they sometimes miscount the letters in a word, and why uncommon words are harder for them.

Key terms

Token
A small piece of text: a word, part of a word or a punctuation mark.
Tokeniser
The tool that splits text into tokens.
Vocabulary
The fixed list of all the tokens a model knows.
Subword
A piece of a word, such as "play" or "ing", used as a token.
Token ID
The number that stands for a token in the vocabulary.
Context window
The largest number of tokens a model can take in at one time.

Explained in detail

Three ways to cut up text

The simplest idea is to make every word a token. The trouble is that a language has hundreds of thousands of words, and new ones arrive all the time. A model that meets a word outside its list, such as a new slang word or an unusual name, has nothing to represent it with. The opposite idea is to make every letter a token. Then nothing is ever unknown, but a short paragraph becomes hundreds of tokens, and single letters carry almost no meaning. Modern systems take a middle path: tokens are pieces of words. Common words stay whole, and rarer words are built from parts.

Method"unhappiness" becomesStrengthWeakness
Whole wordsunhappinessEach token has clear meaningUnknown words cannot be handled
Lettersu, n, h, a, p, p, i, n, e, s, sNothing is ever unknownVery long, with little meaning in each piece
Word piecesun, happi, nessShort, meaningful, and covers new wordsThe pieces do not always match real word parts

How the vocabulary is built

Nobody chooses the pieces by hand. A program starts with single characters and reads a very large amount of text. It finds the pair of neighbouring pieces that occurs most often, such as "t" followed by "h", and joins them into a new piece, "th". It repeats this tens of thousands of times. Frequent combinations grow into whole words, such as "the" and "and", while rare words remain as several smaller pieces. The process stops when the vocabulary reaches the chosen size, typically between thirty thousand and a few hundred thousand pieces.

From token to number to meaning

Each piece in the vocabulary has a position number, its token ID. Text becomes a row of these IDs. An ID by itself is only a label, like a roll number; it says nothing about meaning. So the model's first step is to replace each ID with a long list of numbers that it learned in training, a list that places tokens with related meanings near each other. Everything the model does afterwards is arithmetic on these lists.

Why tokens matter to you

  • Limits: a chatbot's context window is counted in tokens. When a conversation grows beyond it, the earliest parts are dropped and the chatbot "forgets" them.
  • Cost: companies that sell access to language models charge by the token, for what goes in and what comes out.
  • Spelling and counting: a model sees pieces, not letters, so asking it to count letters or reverse a word is harder for it than it looks.
  • Languages: text in a language the tokeniser saw less of is cut into more pieces.

Tokens and Indian languages

Most tokenisers were built mainly from English text. An English sentence of ten words may need about thirteen tokens. The same sentence in Hindi or Tamil can need two or three times as many, because the tokeniser has not learned good pieces for those scripts and falls back on very small ones. More tokens mean higher cost, a context window that fills sooner, and often lower accuracy. Building tokenisers and models with far more Indian-language text is one of the practical steps towards AI that works equally well for Indian users.

How it works

  1. Start with a sentence: "The children are playing happily."
  2. Split it into pieces: The | children | are | play | ing | happi | ly | .
  3. Look up each piece in a fixed list called the vocabulary. Every piece has a number.
  4. The sentence is now a row of numbers, such as 464, 3352, 389, 1514, 278, 9820, 306, 13.
  5. The model works only with these numbers. At the end, numbers are turned back into text.

Why split words into parts?

A language has too many words to list them all, and new words appear every year. So the vocabulary holds common whole words and common word parts. With the parts "play", "ing", "ed" and "er", the model can build playing, played and player without storing each one. A word it has never met can still be assembled from pieces it knows.

TextPossible tokensNote
catcatA common word is one token
unbelievableun | believ | ableA long word becomes parts
Don't!Don | 't | !Punctuation counts as tokens
Kal milte hainK | al | mil | te | hainText the model has seen less of is cut into more pieces

Try it yourself

A sentence becomes numbers

Follow the message "I love mangoes!" into a language model.

Be the tokeniser

You need paper and a pencil. Work alone or with a partner.

  1. Write a small vocabulary with numbers: 1 = the, 2 = cat, 3 = dog, 4 = play, 5 = ing, 6 = ed, 7 = s, 8 = is, 9 = . (full stop).
  2. Turn this sentence into numbers: "The dogs played."
  3. Check: 1, 3, 7, 4, 6, 9.
  4. Give a partner the numbers 1, 2, 8, 4, 5, 9 and ask them to rebuild the sentence.
  5. Now try "The cat is sleeping." What goes wrong?

Think about it

  • Your vocabulary had no piece for "sleep". How could a real tokeniser handle an unknown word?
  • Why is "play" + "ing" more useful than storing "playing" as one piece?

Common mistakes

  • Thinking one token always equals one word. Many words are split into several tokens.
  • Thinking the model reads letters. It reads numbered pieces.
  • Thinking every language is split equally well. Languages with less training text are cut into more, smaller pieces.

Check yourself

Question 1

1. What is a token?

Question 2

2. Why do tokenisers split some words into parts?

Question 3

3. After tokenisation, what does the model actually receive?

Summary

  • Tokenisation splits text into words, word parts and punctuation.
  • Each token becomes a number, and the model works with numbers only.
  • Word parts let a model handle rare and new words, and tokens set its reading limit.

Finished reading?

Sources: Google Machine Learning Crash Course · AI4K12: Five Big Ideas in AI