Choosing features

Feature selection keeps the inputs that help and drops the ones that add noise or leak the answer.

• Drop near-duplicates: two features with correlation 0.98 carry the same information.
• Watch for leakage: features only known after the outcome.
• Model importances or L1 regularisation can rank features.

import numpy as np
a = np.array([1, 2, 3, 4])
b = a * 2
print(round(float(np.corrcoef(a, b)[0, 1]), 2))

1.0

Remember: Ask: would I know this feature at prediction time?

Common mistake: Keeping a 'refund_issued' column to predict returns. It's the answer in disguise.