ML Benchmarks
Benchmarks that measure AI systems on machine learning and computer science work: research engineering, rebuilding published methods, and monitoring other agents.
HackBench
FlawedMeta, RAM team · Whether an LLM monitor can find reward hacking in long agent trajectories
FrontierCS 2
FlawedFrontierCS Team · Whether an agent can rebuild a paper's key contribution and beat the authors' code
FrontierML
FlawedInduction Labs · Agentic ML engineering across 21 tasks and three compute wallets
AI4AI-Bench
FlawedEinsia / Navers Lab · Improving AI training recipes in real research repositories, scored against the shipped recipe
Vals RSI Index
Not enough infoVals AI · Autonomous LLM research across four tasks: compression, LM training, judge harnesses and post-training
How much of each benchmark is affected
Share of checked tasks with a confirmed grading defect
From each review's task log.
Defect rateFlawed threshold, 20%HackBench lower bound: 38% of all 169 records by checks that need no judgement
Rubric results
Failing any check makes a benchmark Flawed. Methodology.
| Benchmark | Open to inspect | Grading | Consistency | Model setup | Fairness |
|---|---|---|---|---|---|
| HackBench | Partial | Flag | Pass | Not reviewed | Pass |
| FrontierCS 2 | Full | Flag | Flag | Pass | Pass |
| DeepRSI | Partial | Flag | Pass | Pass | Pass |
| FrontierML | Full | Flag | Flag | Pass | Pass |
| AI4AI-Bench | Partial | Flag | Pass | Pass | Flag |
| Vals RSI Index | Inadequate | Not reviewed | Not reviewed | Not reviewed | Not reviewed |
More benchmarks coming soon
Math, Reasoning and Multimodal reviews are on the way. Suggest one.