Dropout and regularisation

Dropout randomly switches off some neurons during training, so the network can't rely on any one of them.

• Dropout 0.5 ignores each neuron half the time.
• It acts like training many smaller networks and averaging them.
• It is turned off when making predictions.

import numpy as np
x = np.array([5.0, 6.0, 7.0, 8.0])
mask = np.array([1, 0, 1, 0])
print((x * mask / 0.5).tolist())   # scaled so the average stays the same

[10.0, 0.0, 14.0, 0.0]

Remember: Use dropout when the training score is much higher than validation.

Common mistake: Leaving the model in training mode at prediction time, so outputs change every run.