Model evaluation

Watch, read the card, then practise. ROC and AUC · Imbalanced data · Hyperparameter tuning

Video coming soon

The video for this lesson is being recorded.

Meanwhile, the notes below and the 60-second cards cover the same ideas.

ROC and AUC
An ROC curve shows the trade-off between catching positives and raising false alarms, at every threshold.
Remember: AUC measures ranking, so it doesn't depend on one threshold.
Imbalanced data
When one class is rare, accuracy hides failure. A model can score 99% by never finding the rare case.
Remember: On rare events, look at recall first.
Hyperparameter tuning
Hyperparameters are settings you choose before training, like tree depth or learning rate. Tuning tries options and keeps the best.
Remember: Tune on validation, test once at the very end.

Try the example in Code Lab

tp, fn, fp, tn = 45, 5, 20, 80
print(tp / (tp + fn), fp / (fp + tn))

Lesson 4 of 4

Watch, read the card, then practise. ROC and AUC · Imbalanced data · Hyperparameter tuning

Video coming soon

The video for this lesson is being recorded.

Meanwhile, the notes below and the 60-second cards cover the same ideas.

ROC and AUC
An ROC curve shows the trade-off between catching positives and raising false alarms, at every threshold.
Remember: AUC measures ranking, so it doesn't depend on one threshold.
Imbalanced data
When one class is rare, accuracy hides failure. A model can score 99% by never finding the rare case.
Remember: On rare events, look at recall first.
Hyperparameter tuning
Hyperparameters are settings you choose before training, like tree depth or learning rate. Tuning tries options and keeps the best.
Remember: Tune on validation, test once at the very end.

tp, fn, fp, tn = 45, 5, 20, 80
print(tp / (tp + fn), fp / (fp + tn))