I8 ยท Natural Language Processing in Practice

Tokenisation and Text Cleaning

Tokenisation splits text into units called tokens, which may be words, sub-word pieces or characters. Text cleaning prepares raw text before or alongside tokenisation: removing markup, fixing encoding, lowercasing where appropriate, handling punctuation, numbers, URLs, hashtags and emojis, and optionally removing stop words such as "the" and "is". Modern language models use sub-word tokenisers, such as byte-pair encoding (BPE) or WordPiece, which break rare words into common pieces so that any text can be represented with a fixed vocabulary.

10 min Intermediate ยท Sublevel Advanced Jump to the activity

What you need: The earlier topics of this module.

Definition

"U.S.A.", "don't", "โ‚น1,500", "#MondayMotivation", "kya baat hai!!!" and "๐Ÿ˜‚": a computer has to decide where each word starts and ends before it can do anything else. Getting this wrong quietly ruins everything that follows.

Tokenisation splits text into units called tokens, which may be words, sub-word pieces or characters. Text cleaning prepares raw text before or alongside tokenisation: removing markup, fixing encoding, lowercasing where appropriate, handling punctuation, numbers, URLs, hashtags and emojis, and optionally removing stop words such as "the" and "is". Modern language models use sub-word tokenisers, such as byte-pair encoding (BPE) or WordPiece, which break rare words into common pieces so that any text can be represented with a fixed vocabulary.

In simple words

Before cooking, you wash vegetables and chop them into pieces the right size. Cleaning is the washing; tokenisation is the chopping. Chop too finely and you lose the shape; too coarsely and it will not cook.

Overview

  • Tokenisation splits text into units called tokens, such as words, subwords or characters.
  • Text cleaning removes noise such as HTML tags, extra spaces and unwanted symbols.
  • Common steps are lower-casing, removing punctuation and stop words, and normalising numbers and emojis.
  • Over-cleaning can remove meaning: "not" matters for sentiment.
  • Large language models use subword tokenisers such as byte-pair encoding.
  • Indian languages need Unicode-aware cleaning so that vowel signs are not broken.

Why it matters

Every NLP system starts with tokens. The choices you make, such as whether to lowercase, keep emojis or split "don't", change what a model can learn. Large language models even charge and limit usage by the number of tokens.

Key terms

Word tokenisation
Splitting text into words and punctuation.
Sub-word tokenisation
Splitting rare words into frequent pieces, such as "unhappiness" โ†’ "un", "happi", "ness".
Vocabulary
The set of all tokens a system knows.
Stop words
Very common words such as "the", "is" or "hai" that are sometimes removed.
Regular expression
A pattern language for finding and replacing text, such as all URLs.
Out-of-vocabulary (OOV)
A word the system has never seen and cannot represent with word tokens.

Types

Tokenisers come in several types, and cleaning involves several types of step. Both are covered here.

1. Word tokenisation

Splitting text into words and punctuation marks.

  • How it works: Rules or regular expressions split on spaces and punctuation while handling cases such as "don't", "Dr." and "โ‚น1,200".
  • Example: "I'm in Pune." becoming ["I", "'m", "in", "Pune", "."].
  • When to use it: Use it for classic NLP such as bag-of-words models and n-grams.

2. Sentence tokenisation

Splitting text into sentences.

  • How it works: Full stops, question marks and exclamation marks usually end sentences, but abbreviations such as "Dr." and decimals such as "3.5" must not be split.
  • Example: Splitting a news article into sentences for an extractive summary.
  • When to use it: Use it for summarisation, translation and sentence-level sentiment.

3. Subword tokenisation (BPE, WordPiece, SentencePiece)

Splitting words into frequent pieces learned from data.

  • How it works: Byte-pair encoding repeatedly merges the most frequent pairs of symbols; rare words become several known pieces, so there are no unknown words.
  • Example: "unhappiness" becoming "un", "happi", "ness".
  • When to use it: Use it for transformers and large language models, which all rely on subword vocabularies.

4. Character and byte tokenisation

Treating each character or byte as a token.

  • How it works: The vocabulary is tiny and nothing is unknown, but sequences become long.
  • Example: Spelling-correction models and some models for languages without spaces.
  • When to use it: Use it for noisy text, typos and languages with complex word formation. Byte-level tokenisers in models such as GPT-2 can represent any text, including emojis, without unknown tokens.

5. Text-cleaning steps

Normalising case, removing noise, handling stop words, numbers, URLs and emojis.

  • How it works: Apply only the steps the task needs, using the same steps for training and prediction; for sentiment, keep negations and perhaps emojis.
  • Example: Replacing URLs with <url> and numbers with <num> in tweets before classification.
  • When to use it: Use a documented cleaning function so that results are reproducible.

Explained in detail

Common cleaning steps

StepExampleWhen to be careful
Remove HTML and extra spaces"<b>Great</b> phone" โ†’ "Great phone"Almost always safe
Lowercase"GREAT" โ†’ "great"Capitals can signal names ("Apple") or shouting in reviews
Replace URLs, emails, numbers"call 98xxxxxx" โ†’ "call <NUM>"Numbers matter in prices or dates
Handle emojis"๐Ÿ˜‚" โ†’ "laughing"Emojis carry strong sentiment; do not just delete them
Remove stop words"this is a great phone" โ†’ "great phone""not" is a stop word in many lists, and removing it flips meaning
Normalise spellings"gr8", "greattt" โ†’ "great"Over-correcting can change names and slang

Word, character and sub-word tokens

Word tokens are easy to read but create a huge vocabulary and cannot handle new words: a model trained before 2020 has no token for a new brand name. Character tokens never meet unknown words but make sequences very long and carry little meaning each. Sub-word tokenisers sit in between. Byte-pair encoding, adapted for NLP by Rico Sennrich and colleagues in 2016, starts from characters and repeatedly merges the most frequent adjacent pair until the vocabulary reaches a target size, often 30,000 to 100,000 tokens. Common words become single tokens, while rare words split into pieces. In English, one token is roughly three-quarters of a word on average. For Hindi and other Indian languages, tokenisers trained mainly on English text often split words into many more tokens, which makes processing slower and more expensive; newer tokenisers trained on Indian-language data reduce this gap.

Sentence splitting and special cases

  • A full stop does not always end a sentence: "Dr. Rao arrived at 5 p.m." contains three dots but one sentence.
  • Hindi in Devanagari ends sentences with the danda (เฅค), which a tokeniser built for English may not recognise.
  • Hashtags such as #MondayMotivation pack several words together and may need splitting.
  • Always apply exactly the same cleaning and tokenisation to new text as you did to the training text.

Tokens also matter for cost and limits. Language-model services price requests per token and cap how many tokens fit in one conversation, so the same message written in Hindi may cost two or three times as much as its English translation if the tokeniser splits Devanagari into many small pieces. When you build a product for Indian users, count tokens on real messages in each language before choosing a model, and prefer models whose tokenisers were trained on Indian-language text.

How it works

  1. Inspect raw examples to see what noise exists.
  2. Remove markup and fix encoding problems.
  3. Decide carefully about lowercasing, numbers, emojis and stop words for your task.
  4. Tokenise with a method suited to the language and model.
  5. Save the pipeline so new text is processed identically.

In code

tokenise_clean.py
import re
from collections import Counter

reviews = [
    "<p>Battery is GREAT!!! ๐Ÿ˜‚ Loved it</p>",
    "battery not good, visit https://shop.example for refund",
    "Camera gr8 but battery is not great",
]
STOP = {"is", "it", "for", "but", "the", "a"}           # note: "not" is kept on purpose
EMOJI = {"๐Ÿ˜‚": " laughing "}
SPELL = {"gr8": "great"}

def clean(text):
    text = re.sub(r"<[^>]+>", " ", text)                # remove HTML tags
    text = re.sub(r"https?://\S+", " <URL> ", text)    # replace links
    for e, word in EMOJI.items():
        text = text.replace(e, word)
    return text.lower()

def tokenise(text):
    tokens = re.findall(r"<url>|[a-z0-9]+", text)
    return [SPELL.get(t, t) for t in tokens if t not in STOP]

all_tokens = []
for r in reviews:
    toks = tokenise(clean(r))
    all_tokens += toks
    print(toks)
print("Vocabulary size:", len(set(all_tokens)))
print("Most common:", Counter(all_tokens).most_common(3))

Output

['battery', 'great', 'laughing', 'loved']
['battery', 'not', 'good', 'visit', '<url>', 'refund']
['camera', 'great', 'battery', 'not', 'great']
Vocabulary size: 10
Most common: [('battery', 3), ('great', 3), ('not', 2)]

Try it yourself

The review that flipped

A student cleans phone reviews before training a sentiment model.

Keep or remove for sentiment?

Decide for a review sentiment model.

  • The word "not"

  • The word "terrible"

  • HTML tags such as <br>

  • The emoji ๐Ÿ˜ก

  • Repeated spaces

Common mistakes

  • Removing negation words as stop words.
  • Deleting emojis that carry sentiment.
  • Cleaning training text and new text differently.

Check yourself

Question 1

1. Why do large language models use sub-word tokens?

Question 2

2. Which word should NOT be removed for sentiment analysis?

Question 3

3. "Dr. Rao came at 5 p.m." has how many sentences?

Summary

  • Tokenisation splits text into words, sub-words or characters.
  • Cleaning removes noise but must keep meaning, such as negations and emojis.
  • Sub-word tokenisers such as BPE handle new words; use identical processing for training and new text.

Finished reading?

Sources: Jurafsky and Martin, Speech and Language Processing (3rd ed. draft) ยท scikit-learn: text feature extraction