Natural language processing

Tokenisation

Tokenisation splits text into the pieces a model reads: words, or smaller word pieces.

• Simple: split on spaces and punctuation.
• Modern models use subwords (BPE, WordPiece): 'unhappiness' → 'un', 'happi', 'ness'.
• Subwords mean no word is ever completely unknown.

import re
print(re.findall(r"\w+", "It's 2026, NLP is fun!"))

['It', 's', '2026', 'NLP', 'is', 'fun']

Remember: Models count and charge by tokens, not words.

Common mistake: Assuming one word is one token. Rare words and Hindi words often take several.

Cleaning text

Cleaning makes text consistent before simple models use it: lowercase, trimmed, and without noise.

• Lowercase and strip extra spaces.
• Remove stop words (the, is, and) for bag-of-words models.
• Lemmatisation turns 'studies' and 'studied' into 'study'.

text = "  The Model IS Good  "
stop = {"the", "is"}
print([w for w in text.strip().lower().split() if w not in stop])

['model', 'good']

Remember: Transformers need much less cleaning; they use the full text.

Common mistake: Removing 'not' as a stop word in sentiment analysis, which flips the meaning.

Bag of words and TF-IDF

Bag-of-words counts each word in a document. TF-IDF also lowers the weight of words that appear everywhere.

• TF = count of the word ÷ words in the document.
• IDF = log(documents ÷ documents containing the word).
• Word order is lost: 'dog bites man' = 'man bites dog'.

import math
tf = 2 / 10                  # 2 times in a 10-word document
idf = math.log10(100 / 10)   # in 10 of 100 documents
print(round(tf * idf, 2))

0.2

Remember: TF-IDF + logistic regression is a strong, cheap baseline.

Common mistake: Expecting bag-of-words to understand meaning or word order.

Word embeddings

Embeddings turn words into vectors, so words with similar meanings sit close together.

• word2vec and GloVe give one fixed vector per word.
• Compare vectors with cosine similarity: 1 means the same direction.
• Contextual models (BERT) give a different vector for each use of a word.

import numpy as np
a, b = np.array([1.0, 0.0]), np.array([1.0, 1.0])
print(round(float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b))), 2))

0.71

Remember: Close vectors = similar meaning.

Common mistake: Using static embeddings for words with two meanings, like 'bank'.

Self-attention

In self-attention, every token looks at every other token to decide what's relevant to it.

• Each token makes a query, a key and a value.
• Weights = softmax(Q·Kᵀ ÷ √d); output = weighted mix of values.
• Cost grows with length squared: 512 tokens → 262,144 scores per head.

tokens = 512
print(tokens * tokens)

262144

Remember: Multi-head attention looks for several kinds of relationships at once.

Common mistake: Forgetting positional encodings: attention alone ignores word order.

BERT and GPT

BERT reads text in both directions to understand it; GPT reads left to right to generate it.

• BERT (encoder) is trained by filling in masked words.
• GPT (decoder) is trained to predict the next token.
• Both are pre-trained on huge text, then adapted to tasks.

Example: BERT: 'The [MASK] sat on the mat' → 'cat'. GPT: 'The cat sat on the' → 'mat'.

Remember: Understand or classify → encoder. Write or answer → decoder.

Common mistake: Using an encoder-only model to generate long answers.

Text classification

Text classification gives a whole text a label, like positive or negative, spam or not, or a topic.

• Start simple: TF-IDF features with logistic regression.
• Fine-tune a small transformer when you need more accuracy.
• Multi-label: a text can have several labels at once.

pos, neg = {"good", "love"}, {"bad", "awful"}
words = "i love this phone but the battery is bad".split()
score = sum(w in pos for w in words) - sum(w in neg for w in words)
print("positive" if score > 0 else "negative" if score < 0 else "neutral")

neutral

Remember: Always compare against a simple baseline.

Common mistake: Training on balanced data and deploying on data where 95% is one class.

Named entity recognition

Named entity recognition finds names of people, places, organisations and dates in text.

• It labels each token, so it's a sequence-labelling task.
• BIO tags: B = beginning of an entity, I = inside, O = outside.
• Used for search, form filling and resume parsing.

tokens = ["Infosys", "opened", "in", "Pune"]
tags = ["B-ORG", "O", "O", "B-LOC"]
print([t for t, g in zip(tokens, tags) if g != "O"])

['Infosys', 'Pune']

Remember: B-ORG starts an organisation name; I-ORG continues it.

Common mistake: Tagging each word on its own and splitting 'Tata Consultancy Services' into three entities.