What we do
Benchmarks are the main public way to measure what AI systems can do, and their leaderboards shape what people build, buy and regulate. When a benchmark grades wrongly, its leaderboard can mislead everyone who reads it.
We review public benchmarks against a published rubric. Each review reads the tasks, the graders and, where they are released, the agents' own runs, then says whether the scores can be read the way they are presented. Every finding cites its evidence, and every review ships its task log and analysis scripts.
How reviews are done
Each review follows a written protocol that is published with it. In several reviews, the per-task reading was done by AI model reviewers working under that protocol, and a lead auditor re-checked every defect the verdict relies on against the source. Each review's limitations section says exactly how it was done and what could not be checked.
If a benchmark is fixed, does the verdict change?
A review applies to the version it names. A new version can be reviewed again and gets its own verdict. Reviews stop once the threshold is clearly crossed, so a fixed version may still have other problems.
Are developers told before publication?
Yes. We send each review to the benchmark's developers before we publish it, and we link any response they send.
I disagree with a review
Open an issue on GitHub with the review, the finding and your evidence, or email us. If you develop the benchmark and want your response linked from the review, say so. Corrections are published in the review and noted in its history.
Contact
Email reviews@openbenchmarkreview.org. We read everything; corrections and developer responses come first.