Grading the frontier-model study against results…
Overall accuracy
The external frontier-model study (Gemini, GPT, Claude, Kimi, Llama; 105 fixtures × 3 timings × 3 prompt conditions). Graded against the same ESPN results as the internal pipeline. Imputed rows are statistical estimates (market-anchored by construction) and duplicate rows double-count the same fixture — both are excluded from every metric on this page, as are rows whose match has no stored result yet.
Per model
Winner-pick accuracy by model, all conditions and timings pooled.
Per prompt condition
Accuracy by prompt condition: no external knowledge vs prediction-market odds injected vs external knowledge (web research). "Unparseable" = the condition block was blank in the source (README item 5).
Per timing
Does accuracy improve as the match approaches (24h before → at kickoff → halftime)?
Group vs knockout
Winner-pick accuracy split by stage. Knockout "winner" is the team that advanced (incl. extra time + penalties), so a draw pick there grades as wrong.
Model × condition
The study's core comparison: how each model does under each prompt condition.
Model × timing × condition
The full combination cross table: every model-timing-condition combo with all metrics at once — accuracy, 3-way Brier, ECE, goal accuracy, goal Brier, goal ECE, and the four bet ROIs (match-outcome / goal bets × $1k / $1M bankroll; hover for invested + net). Only eligible collected rows (no imputed/duplicates) on played matches appear.
Market baseline — did the model beat the market?
Each prompt injected the raw Kalshi odds shown below. On every paired fixture (model pick scoreable + market favorite computable), the market favorite is graded the same way: model accuracy vs market accuracy on the identical subset, the delta, how often the model simply echoed the market favorite, and the mean absolute deviation of the model's probabilities from the market's.
Model × condition
Per condition (all models)
3-way Brier score
Multiclass Brier score of each block's 3-way outcome distribution (prob_a / prob_draw / prob_b): BS = Σ(pᵢ − oᵢ)² against the one-hot actual result. Range 0–2, lower is better; random = 0.667. An absent cell grades as probability 0 (the study's blank-cell convention). The market-odds Brier baseline appears in the market-baseline tables above.
Per condition
Confidence calibration
Are high-confidence picks actually more accurate? Each band sums the block's reported confidence (%).
Calibration error (ECE)
Expected Calibration Error of the top-pick confidence: Σ over confidence bands of (band share × |mean confidence − actual accuracy|). 0 = perfectly calibrated; lower is better. Per band, mean confidence vs accuracy shows exactly where a model is over- (mean conf > accuracy) or under-confident.
Per model
Per condition
Per timing
Per band (all models)
Total-goals bucket
Does the max-probability goal bucket match result.total_goals? Coverage is shown first: most models frequently omitted the goal distribution entirely (Claude on ~94% of its real blocks, per the study's README) — accuracy over the covered subset is only meaningful next to its denominator.
Goal-prediction Brier & ECE
Same metrics as above but for the total-goals distribution: Brier over the six goal buckets (0 … 4, 5+; buckets the model left blank count 0, so omitting the actual bucket is penalized) and ECE over the block's dedicated goal confidence (goals_conf). Range 0–2, lower is better. Coverage is sparse by design — Claude omitted goal data on ~94% of its real blocks — so compare Brier only next to its N.
Per condition
Goal ECE per model
Goal Brier per timing
Goal ECE per band (all models)
Bet performance (market odds)
Each condition block's $1,000 / $1,000,000 bankroll allocations graded at the raw Kalshi odds injected into each prompt (the row's own market_odds block): a bet of stake on an outcome priced at implied probability p wins stake × (1/p − 1) if the outcome occurred, loses −stake otherwise; goal buckets are priced from the cumulative Over ladder. Bets without a usable price (knockout rows with no draw cell, rows lacking the ladder) fall back to +stake (even-money). Invested is the sum of actual stakes (some rows exceeded the stated bankroll — kept as-is per the study's README). Toggle the bankroll below (absolute dollars differ; ROI is near-identical because the two allocations are mostly proportional — but not exactly, as the split below shows).
Per model
Per condition
Per timing
ROI by bet kind — match outcome vs goal bets, both bankrolls
The mixed tables above combine the model's match-outcome bets ("X wins"/"Draw") and goal bets ("N goals"). This split shows the two ROIs side by side at both bankroll sizes — independent of the toggle. Hover an ROI cell for invested + net.
Per condition
Per timing
Per-match breakdown
Every fixture in the study. Each model's cell shows the first graded timing's pick (24h → kickoff → halftime); badges explain excluded cells. Fixtures marked "no ground truth" are knockouts the DB has not resolved yet.