Crunching predictions against results…
Overall accuracy
Winner picks compared against ESPN final results across every model, prompt, and timing. "Unparseable" = the model returned empty or non-JSON output (not counted as wrong). "Not played" = matches with predictions but no result yet.
Unparseable outputs taxonomy
What the "Unparseable" tile above actually contains — every prediction in that bucket, classified. Error status = the model/API call failed (timeout, quota, retired model…) so no output exists. Empty output = the call "succeeded" but returned nothing usable. Unparseable content = real output that derive_pick couldn't score. The three top tiles sum to the Unparseable tile.
Categories
Per model
Per prompt
Examples
Per model
Winner-pick accuracy by model. The bar shows correct ÷ comparable.
Per prompt
Accuracy by prompt version.
Per timing
Does accuracy improve as the match approaches (24h before → at kickoff → halftime)?
Group vs knockout
Winner-pick accuracy split by stage. Knockout "winner" is the team that advanced (incl. extra time + penalties).
Model × timing × prompt
The full combination cross table: every model-timing-prompt combo with all metrics at once — accuracy, 3-way Brier, ECE, goal accuracy, goal Brier, goal ECE, and the four bet ROIs (match-outcome / goal bets × $1k / $1M bankroll; hover for invested + net). "Graded" = winner picks scored against a result. Combos with no played-match predictions don't appear.
Confidence calibration
Are high-confidence picks actually more accurate? Each band sums the model's reported outcome_confidence_score (or winner_confidence for stub).
Total-goals bucket
Separate metric: does the max-probability goal bucket ("0 goal" … "5 or more goals") match result.total_goals? "N/A" = no goal bucket emitted (e.g. stub).
3-way Brier score
Multiclass Brier score of each model's 3-way outcome distribution (team A / draw / team B): BS = Σ(pᵢ − oᵢ)² against the one-hot actual result. Range 0–2, lower is better; random = 0.667; a certain-correct pick = 0, a certain-wrong pick = 2. "N/A" = no probability distribution emitted (stub, error, unparseable) or an unresolvable outcome; coverage shows the graded share of predictions on played matches.
By confidence band
Calibration error (ECE)
Expected Calibration Error of the top-pick confidence: Σ over confidence bands of (band share × |mean confidence − actual accuracy|). 0 = perfectly calibrated; lower is better. Composed from the same confidence bands as the table above — per band, mean confidence vs accuracy shows exactly where models are over- (mean conf > accuracy) or under-confident. Note: banded after scale-normalizing, so a confidence that fell back to a 0–1 top-pick probability lands in a (correct) band and may differ from the legacy calibration table above.
Per model
Per timing
Per prompt
Per band (all models)
Goal-prediction Brier & ECE
Same metrics as above but for the total-goals distribution: Brier over the six goal buckets ("0 goal" … "5 or more goals"; buckets the model never emitted count 0, so omitting the actual bucket is penalized), and ECE over the goal-pick's confidence. Range 0–2, lower is better; always predicting "2 or 3 goals" flatly scores ≈ 0.45–0.8 depending on the match. The goal ECE borrows the outcome confidence — these prompts expose one confidence per prediction. "N/A" = no goal distribution emitted (stub, blank rows).
Goal ECE per model
Goal Brier per timing
Goal ECE per band (all models)
Bet performance (market odds)
Each model's bet_value_small_bankroll / bet_value_large_bankroll dollar allocations graded at the market price captured for the bet's timing (24h before / at kickoff / halftime): a bet of stake on an outcome priced at implied probability p wins stake × (1/p − 1) if the outcome occurred, loses −stake otherwise. Odds source per bet: Kalshi → Polymarket for winner bets (the advance market for knockouts), Kalshi → Polymarket totals ladders for goal buckets; bets without a usable price fall back to +stake (even-money). Winner and total-goals allocations are both graded. Toggle the bankroll below (absolute dollars differ; ROI is near-identical because the two allocations are mostly proportional — some rows exceed their bankroll, so not exactly).
Per model
Per prompt
Per timing
Per stage
ROI by bet kind — match outcome vs goal bets, both bankrolls
The mixed tables above combine the model's match-outcome bets ("X wins"/"Draw") and goal bets ("N goal(s)"). This split shows the two ROIs side by side at both bankroll sizes — independent of the toggle. Hover an ROI cell for invested + net.
Per prompt
Per timing
Per stage
Tournament forecasts
Accuracy of the 5 standalone predict_*.py tournament-level forecasts (winner, golden boot, golden ball, reach semifinals, reach final) — each a Polymarket outright market run across ~7 cloud models × 3 prompt variants. Graded against actual WC2026 outcomes stored by tournament_results.py (ESPN winner + golden boot; semifinalists/finalists from ESPN; golden ball = Rodri, Spain). Every stored Prompt Runs run is scored (all snapshots weighted equally). Single-winner tasks score top-1 == actual; reach-market tasks score predicted top-K set == actual set, plus precision-at-K (share of actual qualifiers ranked in the model's top K). "Unparseable" = empty/non-JSON/error output (web-tools prompt has higher rates); "N/A" = ground truth unavailable for that task.
Per task
Per model
Per prompt variant
Per snapshot stage
Reach-market precision-at-K
Per-run breakdown
Per-match breakdown
Every match the pipeline predicted on. Expand a match to see each model's pick, confidence, and whether it was right.