Train/Test Split and Out-of-Sample Testing

## Why In-Sample Testing Is Misleading If you fit a model on a dataset and then evaluate it on the same dataset, you will always get an overly optimistic result. The model has "seen" the answers and has been optimised for them. This is in-sample evaluation — it tells you how well the model memorised the training data, not how well it predicts the future. ## The Train/Test Split The solution: reserve a portion of data exclusively for testing. The model never sees this data during training. Standard split: - 70–80% of data: training set (model fitted on this) - 20–30% of data: test set (model evaluated on this) For time-series data (sports matches are time-ordered), use a temporal split: all matches before date X for training, all matches after date X for testing. Do not randomly shuffle time-series data — this introduces lookahead bias. ## Walk-Forward Testing Even a single temporal split can be misleading if the split date is chosen to favour the model. Walk-forward testing (also called rolling window validation) is more robust: 1. Train on months 1–12. Test on month 13. 2. Train on months 2–13. Test on month 14. 3. Continue rolling forward. 4. Aggregate test results across all periods. This simulates how the model would perform in genuine real-time deployment. ## The Minimum Test Set Size A test set must be large enough to provide statistically meaningful evaluation. For a betting model: - Minimum test set: 300 matches per league - Preferred: 500+ matches per league - Multiple seasons preferred over a single season (captures different competitive environments)