Suggest a benchmark
Open an issue on GitHub titled with the benchmark's name, or email reviews@openbenchmarkreview.org. Tell us:
- Where the tasks, graders and leaderboard live, with links.
- Which version you mean: a commit, dataset revision or leaderboard date.
- What made you suspicious, if anything: a run that passed too easily, a correct answer marked wrong, a model configured differently from the rest.
We prioritise benchmarks that are widely cited and whose tasks and runs are public enough to review.
Contribute a review
Reviews are written in the open. To contribute one, open a pull request on GitHub that adds a folder for the benchmark with:
- A report written from the report template, following the methodology.
- A task log with one row per inspected task, its classification and its evidence.
- A setup checklist recording every leaderboard entry's model, harness and settings.
- The scripts that fetch the public data and reproduce your counts.
A second reviewer re-checks every counted defect before a review is published under the project's name.
For benchmark developers
If we have reviewed your benchmark, you can send a response to be linked from the review, a correction with evidence, or a new version for re-review. Email reviews@openbenchmarkreview.org or open an issue. If you want private access reviewed under NDA rather than public release, say so.