Lesson 1 of 4

Text preprocessing

Watch, read the card, then practise. Tokenisation · Cleaning text

Video coming soon

The video for this lesson is being recorded.

Meanwhile, the notes below and the 60-second cards cover the same ideas.

Tokenisation
Tokenisation splits text into the pieces a model reads: words, or smaller word pieces.
Remember: Models count and charge by tokens, not words.
Cleaning text
Cleaning makes text consistent before simple models use it: lowercase, trimmed, and without noise.
Remember: Transformers need much less cleaning; they use the full text.

Try the example in Code Lab

import re
print(re.findall(r"\w+", "It's 2026, NLP is fun!"))

Lesson 1 of 4

Watch, read the card, then practise. Tokenisation · Cleaning text

Video coming soon

The video for this lesson is being recorded.

Meanwhile, the notes below and the 60-second cards cover the same ideas.

Tokenisation
Tokenisation splits text into the pieces a model reads: words, or smaller word pieces.
Remember: Models count and charge by tokens, not words.
Cleaning text
Cleaning makes text consistent before simple models use it: lowercase, trimmed, and without noise.
Remember: Transformers need much less cleaning; they use the full text.

import re
print(re.findall(r"\w+", "It's 2026, NLP is fun!"))