Tokenisation

Tokenisation splits text into the pieces a model reads: words, or smaller word pieces.

• Simple: split on spaces and punctuation.
• Modern models use subwords (BPE, WordPiece): 'unhappiness' → 'un', 'happi', 'ness'.
• Subwords mean no word is ever completely unknown.

import re
print(re.findall(r"\w+", "It's 2026, NLP is fun!"))

['It', 's', '2026', 'NLP', 'is', 'fun']

Remember: Models count and charge by tokens, not words.

Common mistake: Assuming one word is one token. Rare words and Hindi words often take several.