Adam Allcock, 2026
📄 Read the paper (PDF) · 
Public LLM leaderboards rank models to the tenth of a percent — against answer keys that are themselves wrong. Auditing four of the most-quoted multiple-choice benchmarks under one taxonomy (wrong keys, malformed items, and questions with more than one defensible answer), 4.5–11.8% of items can't fairly score a model, against a near-zero rate on a saturated MATH-500 control.
We don't just diagnose — we ship the fix: corrected drop-in "-Clean" releases of all four, with per-item confidence tiers, the full flagged-candidate ledger, and dual original-vs-cleaned scoring, so an existing eval harness rescores with no code changes.
Two results travel beyond these sets: our model-adjudicated verdicts track GPQA's own recorded expert-disagreement (external validation, not circular), and the standard single-pass disagreement method undercounts the rate by 2.3×. An audit is itself a measurement instrument, so we calibrate ours against injected defects and never-flagged controls and report rates as bands, not points. Model-adjudicated and calibrated — not human-certified.
| Release | Source | Cleaned | License |
|---|---|---|---|
| gpqa-diamond-clean | GPQA Diamond | 198 → 189 | CC BY 4.0 |
| gpqa-ext-complement-clean | GPQA Extended (minus Diamond) | 348 → 309 | CC BY 4.0 |
| mmlu-pro-clean | MMLU-Pro | 12,032 → 10,689 | MIT |
| mmmu-pro-clean | MMMU-Pro (standard, 10-option) | 1,730 → 1,526 | Apache-2.0 |
Each is a drop-in: a harness that runs the original runs the -Clean set unchanged, and every removal is auditable item by item.
Allcock, A. (2026). When the Answer Key Is Wrong: A Calibrated Audit of GPQA, MMLU-Pro, and MMMU-Pro, with Corrected Releases. Zenodo. https://doi.org/10.5281/zenodo.21613590
@misc{allcock2026answerkey,
title = {When the Answer Key Is Wrong: A Calibrated Audit of GPQA, MMLU-Pro, and MMMU-Pro, with Corrected Releases},
author = {Allcock, Adam},
year = {2026},
doi = {10.5281/zenodo.21613590},
publisher = {Zenodo}
}