AI4AI-Bench

AI4AI-Bench asks whether coding agents can improve an existing AI training recipe. Its ten tasks span diffusion, alignment, reasoning distillation, unlearning, pruning, RL, reward modelling, graph diffusion and model merging. An agent explores for 4 hours; only its code change is replayed in a fresh 12-hour training run, and the best of up to three checkpoints is scored on held-out data and mapped to 0–1, where 0.1 is the shipped recipe. Six systems at several reasoning efforts give 29 configurations and 290 single-run cells.¹

What we found

• 4 of the 10 tasks (40%) have confirmed grading defects. A benchmark is rated Flawed at 20%.
• In the Sokoban task, the "hidden" test boards come from the upstream project's default seed, which sits in code the agent edits, and training covered most of the test boards; five cells score a perfect 512/512.
• The image task scores images that have collapsed in prompt-following and variety, because the safety checks its README promises do not exist; its "hidden" test seed is also written in code the agent can read.
• One task's declared best possible score is twice its metric's maximum, and another's baseline sits inside seed noise.
• Every task reports the best of three checkpoints, picked on the test set.
• Each configuration runs once, so differences of about 0.1 between configurations can't be separated from noise.

Does the ranking hold? Partly. Opus 5 is first when results are averaged over effort levels, but third when systems are compared at maximum effort; with the confirmed corrections, Kimi K3 moves from third to second. No cell was judged research, and 49% gained nothing over the shipped recipe.

Representative Errors

• A held-out seed that is not hidden. In ragen_sokoban_grpo, declaration.py:50 sets FINAL_ENVIRONMENT_SEED = 123, described as "host-side so exploration cannot reconstruct the held-out boards". Upstream RAGEN's config/base.yaml sets seed: val: 123, and that tree is copied into the agent's editable /workspace/ragen. Generation is deterministic, so the final bank, solutions included, can be rebuilt in the container. No agent was seen doing so.³ Separately, training on solver demonstrations covered 243–490 of the 512 final boards, and five cells score a perfect 512/512. One of those got there by pure imitation, with no RL. The proxy summary also hard-codes its overlap with the final set as 0.0, although 7 boards overlap.
• Scorer optimisation counted as success. The ddpo_sd15_aesthetic README promises "alignment/diversity gates", but the evaluator says "the original DDPO benchmark has no alignment/diversity guard" (final_eval.py:233). Eight cells either back-propagate through the frozen aesthetic scorer or over-train PPO. Their images collapse in prompt alignment and diversity, yet they score 6.8–17.7 against a shipped 5.53 and supply 94% of the task's above-baseline σ. The "score-only" final seed (20269702) is also hard-coded in the mock path of /opt/harness/final_eval.py:261, which the agent can read.
• An optimum the metric cannot reach. In openunlearning_tofu_npo_llama3p2_1b, the balanced score is the harmonic mean of two values clipped to [0, 1] (final_eval.py:192-221), so its maximum is 1.0. The paper anchors the optimum at 2.08, which caps σ at 0.185. The six cells that reach the true maximum are scored as covering 7.5% of the distance to the optimum.
• A baseline inside the noise. In ultrafeedback_bt_rm_rewardbench, the task calls its own reference "seed-sensitive … a single-run reference" (74.569), and the paper anchors at 74.9. Replays of the unchanged recipe score 65.4–73.95. The official score also includes the 512 proxy pairs the agent saw while exploring.
• Checkpoints chosen on the test set. orchestrator/final_score.py:148 reports the best of up to three checkpoints on the final metric, and 163 of 271 completed cells report a non-final one. Several agents published three seeds as their three checkpoints, so the reported score is a max over seeds. Using the last checkpoint instead flips 21 cells between beating and missing the baseline.

Errors

Sensitivity of the ranking

Re-ranking the six systems by mean σ under different rules:

Each cell is a single run, and effort curves are non-monotone: Opus 5 scores 0.287 at medium and 0.188 at xhigh. Gaps of about 0.1 σ between configurations are therefore within noise. Under the paper's aggregation, a task bootstrap puts Opus 5 at #1 in 98% of replicates, and splits #2 between Sol (59%) and Kimi K3 (39%).

Rubric

1. Reviewability

NotesEvery task contract, evaluator and orchestrator is public, along with 290 case pages carrying checkpoint scores and per-cell audits. Patch contents and raw logs are not published, despite the paper's claim. Neither is the scorer behind the published OpenR1 numbers, nor the harnesses' token and context settings.

2. Scoring

3. Evaluation quality

Disclaimer

This is a full review of all 10 tasks and 290 cells. AI4AI-Bench publishes no raw logs or patches, so cell-level calls rest on the benchmark's own case pages. Wherever a page's claim could be checked against the code, we checked it. No agent was observed using the leaked RAGEN or DDPO seeds, so their effect on the observed scores is unproven.

Notes

1. Paper: arxiv.org/abs/2608.20318 (v1, Aug. 20, 2026). Homepage: lab.einsia.ai/ai4ai.
2. Repository: github.com/Einsia/AI4AI-Bench at 5ced1e2. Case pages: lab.einsia.ai/ai4ai/trajectories, fetched Oct. 29, 2026.
3. Upstream RAGEN: github.com/RAGEN-AI/RAGEN at 20daedc4, config/base.yaml lines 8–10: seed: train: 10000, val: 123.
4. Per-task notes, per-cell reviews, the adjudication, the task log, the setup audit and the scripts are in audits/ai4ai-bench.

Cite