Calibration Curves and Reliability Diagrams

## What Is Calibration? A model is perfectly calibrated if, for all matches where it predicts 60% home win probability, the home team wins 60% of the time. In reality, most models are miscalibrated in specific ranges. ## The Calibration Curve To build a calibration curve (reliability diagram): 1. Group all predictions into probability bins (0–10%, 10–20%, ..., 90–100%) 2. For each bin: calculate average predicted probability and actual outcome frequency 3. Plot predicted probability (x-axis) vs actual frequency (y-axis) A perfectly calibrated model lies on the diagonal (y = x line). Points above the diagonal: model underestimates probability. Points below: model overestimates. ## Common Miscalibration Patterns **Overconfidence:** The calibration curve is flatter than the diagonal — predictions of 80% correspond to 65% actual frequency. The model is too confident. **Underconfidence:** The calibration curve is steeper than the diagonal — predictions of 80% correspond to 90% actual frequency. The model is too conservative. **Range miscalibration:** The model is well-calibrated in the 40–60% range but poorly calibrated at extremes. Common in models fitted primarily on "competitive" matches. ## Fixing Miscalibration **Platt scaling:** Fit a logistic regression on the model's raw output vs outcomes. Use the logistic regression's output as the final probability. **Isotonic regression:** A more flexible, non-parametric calibration method. Preferred when calibration errors are non-monotonic. **Beta calibration:** Uses the beta distribution to calibrate probability outputs. Best theoretical fit for probability-valued outputs. After applying calibration, rebuild the calibration curve and confirm improvement.