Overfitting: Recognition and Prevention

## The Overfitting Problem Overfitting occurs when a model learns to predict the training data too well — including the noise, not just the signal. An overfitted model performs well in sample but poorly out of sample. ## Signs of Overfitting - Training performance significantly better than test performance - Model has many parameters relative to training data size - Removing or adding individual training examples significantly changes predictions - Calibration on training data is good; calibration on test data is poor ## The Bias-Variance Tradeoff All models face the bias-variance tradeoff: - **High bias (underfitting):** Model is too simple to capture the true pattern. High error on both training and test data. - **High variance (overfitting):** Model is too complex, fitting noise. Low training error, high test error. - **Optimal:** Model captures the true signal without fitting the noise. ## Prevention Strategies **1. Regularisation:** Add a penalty for complexity to the model's loss function. L2 regularisation (Ridge): shrinks all coefficients toward zero. L1 regularisation (Lasso): forces some coefficients to exactly zero (feature selection). **2. Early stopping:** In iterative models: stop training when test performance stops improving (even if training performance continues improving). **3. Feature reduction:** Fewer features = lower overfitting risk. Use domain knowledge to select theoretically motivated features. **4. Cross-validation:** Rigorous time-series cross-validation reveals overfitting — models that overfit show large train/test performance gaps. **5. Minimum sample per feature:** Rule of thumb: at least 50–100 observations per model parameter. A 10-feature model needs 500–1,000 training matches.