Live Model Monitoring and Drift Detection

## Models Degrade Over Time A model fitted on historical data reflects the world as it was when that data was collected. As the sport evolves — tactical innovations, rule changes, player quality changes — the model may become less accurate. This degradation is called model drift. ## Types of Model Drift in Sports **Concept drift:** The underlying relationship between features and outcomes changes. Example: the introduction of VAR changed the probability of certain match outcomes (more late decisions, fewer "wrong" goals stand). **Data drift:** The distribution of input features changes. Example: the average goals per match in a league changes over multiple seasons due to tactical evolution. A model trained on 3.0 goals/match average will underestimate goals in a 2.6 goals/match era. **Covariate shift:** The types of matches in the market change. Example: expansion to a new league with different characteristics. ## Drift Detection Methods **Performance monitoring:** Track log-loss and CLV on a rolling 100-match window. A sustained deterioration is evidence of drift. **Feature distribution monitoring:** Track the distribution of key features (average goals per match, home win rate) over time. Significant shifts indicate data drift. **Residual analysis:** Track model residuals (predicted − actual) over time. If residuals show a systematic trend (consistently positive or negative in a specific period), the model is miscalibrated for current conditions. ## The Response Protocol On detection of significant drift: 1. Identify the source (concept drift vs data drift) 2. Update the training data window (remove oldest data, add recent) 3. Re-estimate model parameters 4. Evaluate on most recent held-out data 5. Deploy updated model