Machine learning: intermediate and advanced

One 60-second card per skill, in learning order. 11 of 11 cards ready.

Topic 1

Trees and ensembles

Decision trees · Random forests · Gradient boosting

Open the lesson Practise this topic

60-second card

Decision trees

A decision tree predicts by asking yes/no questions about the features until it reaches an answer (a leaf).

Each split picks the question that best separates the classes (lowest Gini impurity or entropy).
Gini = 1 − sum of squared class shares; 0 means the node is pure.
Trees don't need feature scaling and are easy to explain.

shares = [0.8, 0.2]          # 8 'yes', 2 'no'
print(round(1 - sum(s * s for s in shares), 2))

Prints

0.32

Remember: Limit max_depth, or the tree will memorise the training data.

Common mistake: Letting a tree grow fully, then wondering why the test score is much lower.

60-second card

Random forests

A random forest trains many decision trees on random samples and combines their votes.

Bagging: each tree gets a random sample of rows, drawn with replacement.
Each split only looks at a random subset of features (max_features).
Averaging many different trees cancels out their individual mistakes.

votes = ["fraud", "ok", "fraud", "fraud", "ok"]
print(max(set(votes), key=votes.count))

Prints

fraud

Remember: Many varied trees beat one deep tree.

Common mistake: Expecting more trees to cause overfitting. Extra trees mostly just cost time.

60-second card

Gradient boosting

Gradient boosting adds trees one at a time, each one fixing the errors left by the trees before it.

Each new tree learns the remaining errors (residuals).
A smaller learning rate needs more trees but often generalises better.
XGBoost, LightGBM and CatBoost are popular for tabular data.

target, prediction = 100.0, 0.0
for step in range(3):
    prediction += 0.5 * (target - prediction)   # each tree fixes half the error
print(prediction)

Prints

87.5

Remember: Use early stopping on a validation set to pick the number of trees.

Common mistake: Adding rounds until training loss is tiny. Watch validation loss instead.

Topic 2

Feature engineering

Scaling · Encoding categories · Choosing features

Open the lesson Practise this topic

60-second card

Scaling

Scaling puts features on similar ranges, so no feature dominates just because of its units.

Min-max: (x − min) ÷ (max − min) gives 0 to 1.
Standardising: (x − mean) ÷ std gives a z-score.
Matters for kNN, SVMs, linear models and neural networks; not for trees.

import numpy as np
x = np.array([10, 30, 60])
print(((x - x.min()) / (x.max() - x.min())).round(2).tolist())

Prints

[0.0, 0.4, 1.0]

Remember: Fit the scaler on training data only, then apply it to test data.

Common mistake: Fitting the scaler on all data, which leaks test information into training.

60-second card

Encoding categories

Models need numbers, so text categories must be encoded.

One-hot: one 0/1 column per category. Good when there's no order.
Ordinal: 0, 1, 2 for ordered values like low, medium, high.
Thousands of categories (like pincodes)? Group them or use target encoding.

cities = ["Delhi", "Mumbai", "Delhi"]
names = sorted(set(cities))
print(names, [[int(c == n) for n in names] for c in cities])

Prints

['Delhi', 'Mumbai'] [[1, 0], [0, 1], [1, 0]]

Remember: No natural order → one-hot. Real order → ordinal.

Common mistake: Numbering cities 0, 1, 2, which tells the model Chennai > Mumbai > Delhi.

60-second card

Choosing features

Feature selection keeps the inputs that help and drops the ones that add noise or leak the answer.

Drop near-duplicates: two features with correlation 0.98 carry the same information.
Watch for leakage: features only known after the outcome.
Model importances or L1 regularisation can rank features.

import numpy as np
a = np.array([1, 2, 3, 4])
b = a * 2
print(round(float(np.corrcoef(a, b)[0, 1]), 2))

Prints

1.0

Remember: Ask: would I know this feature at prediction time?

Common mistake: Keeping a 'refund_issued' column to predict returns. It's the answer in disguise.

Topic 3

Unsupervised learning

Clustering · Dimensionality reduction

Open the lesson Practise this topic

60-second card

Clustering

Clustering groups similar points without labels. k-means finds k groups around centres.

Assign each point to the nearest centre.
Move each centre to the mean of its points. Repeat until stable.
Choose k with the elbow method or silhouette score.

import numpy as np
cluster = np.array([[1, 2], [3, 4], [5, 6]])
print(cluster.mean(axis=0).tolist())

Prints

[3.0, 4.0]

Remember: Scale features first: clustering uses distances.

Common mistake: Using k-means on ring or curved shapes. Try DBSCAN for those.

60-second card

Dimensionality reduction

Dimensionality reduction squeezes many features into a few, keeping most of the information.

PCA finds new directions (components) ordered by how much variance they keep.
Keep enough components for, say, 90% of the variance.
t-SNE and UMAP are for 2D pictures, not for model inputs.

variance = [4.0, 3.0, 2.0, 1.0]
print(f"{(variance[0] + variance[1]) / sum(variance):.0%}")

Prints

70%

Remember: Standardise before PCA, or big-unit features dominate.

Common mistake: Reading meaning into t-SNE distances between far-apart clusters.

Topic 4

Model evaluation

ROC and AUC · Imbalanced data · Hyperparameter tuning

Open the lesson Practise this topic

60-second card

ROC and AUC

An ROC curve shows the trade-off between catching positives and raising false alarms, at every threshold.

True positive rate (recall) = TP ÷ (TP + FN).
False positive rate = FP ÷ (FP + TN).
AUC 0.5 is random guessing; 1.0 is perfect ranking.

tp, fn, fp, tn = 45, 5, 20, 80
print(tp / (tp + fn), fp / (fp + tn))

Prints

0.9 0.2

Remember: AUC measures ranking, so it doesn't depend on one threshold.

Common mistake: Comparing models by accuracy at 0.5 when the right threshold depends on the cost of errors.

60-second card

Imbalanced data

When one class is rare, accuracy hides failure. A model can score 99% by never finding the rare case.

Measure precision, recall and F1, or PR-AUC.
Use class weights, or oversample the rare class (e.g. SMOTE).
Split with stratify, so both sets keep the same class shares.

labels = [0] * 990 + [1] * 10
always_no = [0] * 1000
print(sum(p == y for p, y in zip(always_no, labels)) / 1000)

Prints

0.99

Remember: On rare events, look at recall first.

Common mistake: Celebrating 99% accuracy on fraud data that's 1% fraud.

60-second card

Hyperparameter tuning

Hyperparameters are settings you choose before training, like tree depth or learning rate. Tuning tries options and keeps the best.

Grid search tries every combination; random search samples them.
Score each option with cross-validation, not the test set.
Fits = combinations × folds, so grids grow fast.

depths, trees, folds = [3, 5, 8], [100, 200, 300, 500], 5
print(len(depths) * len(trees) * folds)

Prints

Remember: Tune on validation, test once at the very end.

Common mistake: Picking settings by test score, which makes the test result too optimistic.