Loss functions

The loss measures how wrong the predictions are. Training tries to make it smaller.

• Regression: mean squared error (MSE) or mean absolute error (MAE).
• Classification: cross-entropy, −log(probability of the right class).
• Confident wrong answers get a large cross-entropy loss.

import math
print(round(-math.log(0.9), 2), round(-math.log(0.1), 2))

0.11 2.3

Remember: Number → MSE. Category → cross-entropy.

Common mistake: Using accuracy as the loss. It can't be trained with gradients.