Machine learning basics

Supervised or unsupervised

Supervised learning learns from examples with answers; unsupervised learning finds patterns without them.

• Supervised: past loans marked 'repaid' or 'defaulted' → predict new ones.
• Regression predicts a number; classification predicts a category.
• Unsupervised: group similar customers with no labels (clustering).

Example: Predict house price → regression. Spam or not → classification. Find customer groups → clustering.

Remember: Have the answers in your data? Supervised. Don't? Unsupervised.

Common mistake: Calling price prediction 'classification'. A number means regression.

Features and labels

Features are the inputs a model uses; the label is what it predicts.

• To predict marks from hours studied and attendance: hours and attendance are features.
• Marks is the label (also called the target).
• In code, features are often X and the label is y.

Example: Row: hours = 5, attendance = 90%, marks = 78. Features (5, 90%), label 78.

Remember: X = inputs, y = the answer.

Common mistake: Using a feature that's only known after the answer, like 'refund issued' to predict returns.

Training and test data

Split your data: train the model on one part and test it on data it has never seen.

• A common split is 80% train, 20% test.
• The test score estimates how it will do on new data.
• Shuffle before splitting, and keep a fixed random seed.

rows = list(range(100))
cut = int(len(rows) * 0.8)
train, test = rows[:cut], rows[cut:]
print(len(train), len(test))

80 20

Remember: Never let the model see the test set while learning.

Common mistake: Scoring the model on its own training data. That only shows how well it memorised.

Linear regression

Linear regression fits a straight line: prediction = weight × feature + bias.

• The weight (slope) says how much the prediction changes per unit of the feature.
• The bias (intercept) is the prediction when the feature is 0.
• Training picks the line with the smallest squared errors.

w, b = 5.0, 40.0          # marks = 5 × hours + 40
hours = 6
print(w * hours + b)

70.0

Remember: One feature: y = m·x + c. More features: one weight each.

Common mistake: Trusting the line far outside the data you trained on.

MAE, MSE and RMSE

MAE, MSE and RMSE measure how far predictions are from the real numbers.

• MAE: average of the absolute errors. Easy to read.
• MSE: average of squared errors. Punishes big misses.
• RMSE: square root of MSE, back in the original units.

actual = [100, 200, 300]
pred = [110, 190, 330]
errors = [p - a for a, p in zip(actual, pred)]
mae = sum(abs(e) for e in errors) / 3
mse = sum(e * e for e in errors) / 3
print(round(mae, 2), round(mse, 2), round(mse ** 0.5, 2))

16.67 366.67 19.15

Remember: Lower is better for all three.

Common mistake: Comparing an MSE with an MAE directly. They're on different scales.

Classification models

A classifier predicts a category, often by first giving a probability and then applying a threshold.

• Logistic regression gives a probability between 0 and 1.
• With a 0.5 threshold, 0.7 → 'yes', 0.3 → 'no'.
• Other classifiers: decision trees, k-nearest neighbours, random forests.

probs = [0.9, 0.4, 0.65, 0.2]
print(["spam" if p >= 0.5 else "not spam" for p in probs])

['spam', 'not spam', 'spam', 'not spam']

Remember: Probability first, then a threshold decides the class.

Common mistake: Assuming 0.5 is always the right threshold. It depends on the cost of each mistake.

Accuracy and the confusion matrix

A confusion matrix counts the four kinds of right and wrong answers.

• TP: said yes, was yes. TN: said no, was no.
• FP: said yes, was no (false alarm). FN: said no, was yes (missed).
• Accuracy = (TP + TN) ÷ all.

tp, fp, fn, tn = 40, 10, 20, 30
print((tp + tn) / (tp + fp + fn + tn))

0.7

Remember: Accuracy can mislead when one class is rare.

Common mistake: Reporting 99% accuracy on data that's 99% 'no'. Always saying 'no' gets that too.

Precision and recall

Precision asks 'of my alarms, how many were real?' Recall asks 'of the real ones, how many did I catch?'

• Precision = TP ÷ (TP + FP).
• Recall = TP ÷ (TP + FN).
• F1 combines both into one number.

tp, fp, fn = 40, 10, 20
p, r = tp / (tp + fp), tp / (tp + fn)
print(round(p, 2), round(r, 2), round(2 * p * r / (p + r), 2))

0.8 0.67 0.73

Remember: Missing a disease is costly → care about recall. Annoying false alarms → precision.

Common mistake: Raising one and forgetting the other usually drops.

Overfitting and underfitting

Overfitting is memorising the training data; underfitting is not learning enough from it.

• Overfit: great on training data, poor on test data.
• Underfit: poor on both.
• Fixes for overfitting: more data, a simpler model, regularisation.

Example: Train 99%, test 70% → overfitting. Train 65%, test 63% → underfitting.

Remember: Compare training and test scores. A big gap means overfitting.

Common mistake: Making the model more complex when it's already overfitting.

Cross-validation

Cross-validation tests a model several times on different slices of the data, for a fairer score.

• 5-fold: split into 5 parts, train on 4, test on 1, and repeat 5 times.
• Average the 5 scores.
• Use it to compare models and settings without touching the test set.

scores = [0.82, 0.85, 0.80, 0.84, 0.79]
print(round(sum(scores) / len(scores), 2))

0.82

Remember: More reliable than one split, especially with small data.

Common mistake: Choosing settings by test-set score. Use cross-validation, then test once at the end.

Regularisation

Regularisation adds a penalty for large weights, which keeps a model simpler and stops overfitting.

• L2 (ridge) shrinks all weights a little.
• L1 (lasso) can push some weights to exactly 0, dropping features.
• A stronger penalty means a simpler model.

Example: Loss = error + λ × (sum of squared weights). Bigger λ → smaller weights.

Remember: Too much penalty underfits; too little overfits. Tune it.

Common mistake: Regularising without scaling features first, so big-unit features get punished unfairly.