## The Model Comparison Framework
At some point, you will have multiple candidate models: Poisson regression, Elo ratings, logistic regression, machine learning. How do you decide which to use?
## The Comparison Metrics
Evaluate each candidate on the same held-out test set. Compare:
1. **Log-loss:** Which model produces the best-calibrated probability estimates?
2. **CLV against Pinnacle closing:** Which model's bets would have had the highest average CLV?
3. **Simulated ROI:** Which model would have produced the highest ROI in backtesting?
4. **Stability:** Does the model's performance hold across different seasons and competitions?
## The Statistical Significance of Differences
A difference in log-loss of 0.002 between two models on 500 matches is unlikely to be meaningful. Calculate whether the difference is statistically significant before concluding one model is superior.
Permutation test: randomly shuffle model A vs model B predictions 1,000 times and calculate the log-loss difference distribution. If the observed difference falls outside the 95th percentile of the shuffled distribution: the difference is statistically significant.
## The Ensemble Option
If multiple models each have genuine but distinct predictive information, combining them (ensemble) often outperforms any single model:
Combined probability = w₁ × P_model1 + w₂ × P_model2 + ... (weights sum to 1)
Optimal weights can be estimated by minimising log-loss on a validation set.
## The Parsimony Principle
When two models produce similar performance: choose the simpler one. Simpler models are easier to maintain, easier to debug when they fail, and less likely to have overfitted subtle patterns in historical data.
Create a free account to track your progress and save bookmarks.
