Experiment tracking

Experiment tracking records every training run: settings, data version, metrics and outputs.

• Tools: MLflow, Weights & Biases.
• Fix random seeds so runs can be repeated.
• Pick the best run by validation score, then test once.

runs = [{"id": 1, "lr": 0.1, "val": 0.84}, {"id": 2, "lr": 0.01, "val": 0.89}, {"id": 3, "lr": 0.001, "val": 0.86}]
best = max(runs, key=lambda r: r["val"])
print(best["id"], best["lr"])

2 0.01

Remember: If it isn't logged, you can't reproduce it.

Common mistake: Choosing the run with the best training accuracy.