Hyperparameter tuning

Hyperparameters are settings you choose before training, like tree depth or learning rate. Tuning tries options and keeps the best.

• Grid search tries every combination; random search samples them.
• Score each option with cross-validation, not the test set.
• Fits = combinations × folds, so grids grow fast.

depths, trees, folds = [3, 5, 8], [100, 200, 300, 500], 5
print(len(depths) * len(trees) * folds)

60

Remember: Tune on validation, test once at the very end.

Common mistake: Picking settings by test score, which makes the test result too optimistic.