FrontierML

FrontierML is an agentic benchmark of 21 ML tasks: training, distillation, quantization, serving, vision, audio, Lean proofs and games. Each model works for 12 hours in its developer's own harness, with internet access and a compute wallet of $10, $50 or $200, once per wallet. A held-out metric is scaled between a per-task floor and ceiling to 0–100. A Claude Opus 5.5 reviewer can zero trials that break the rules. The leaderboard reports the mean per wallet.¹

What we found

• 7 of the 21 tasks (33%) have confirmed grading defects. A benchmark is rated Flawed at 20%.
• The hidden test data for all 21 tasks is in a public Hugging Face dataset, and the agents have internet access and a Hugging Face token.
• The review that zeroes rule-breaking runs was switched off for 38 runs, 21 of them DeepSeek's, and those scores still count.
• Each model runs once per task and budget. Near-identical runs differ by a median of 5.2 points, so several budget winners are within noise.

Does the ranking hold? Yes. Opus 5.5 ranks first in every resample of the tasks. The score levels don't hold, and several tasks measure something other than what they claim.

Representative Errors

• Held-out answers readable from the web. Every verifier downloads its held-out data from <task>/private/ in inductionlabs/frontier-ml-tasks. That dataset is public and ungated, and was created on Sept. 24, before any audited run. Agents have internet access, and by default they receive the operator's own HF token (harbor_plugin.py:521-526). The only control is the reviewer: 10 of 21 rubrics forbid using the dataset, and the "searching up answers" check the methodology post describes appears in no rubric. DeepSeek 4ttc3o cloned the benchmark repository, read the grader and the reference solution, and learned where the private split lives. It downloaded nothing and scored 0, and its review had been skipped.
• Selecting on the graded benchmark, scored as valid. The prompt bans training, fine-tuning, calibrating "or selecting" on any public benchmark's test split. Graded GPQA-Diamond and LiveCodeBench files ship to the agent as development data. In ternary-ptq, the rubric omits "select", so it passed three Fable trials and one Argon trial that picked settings or checkpoints by GPQA accuracy. One is Fable 27dhfi: "GPQA-Diamond first 40 questions: 4/40…", after which it changes its shipped defaults to match. In llm-post-train, the same behaviour was zeroed twice and passed once (Fable 2ikiw0).
• A scorer that a blind constant beats. In screen-difference, the floor is an empty output. An input-blind fixed paragraph scores 32.0, above 14 of the 18 audited trials. The judge scores recall only, although the prompt forbids unsupported claims.
• A rule that cannot decide its own winner. In llm-diffusion, the review separates about 80 points from 0 on "genuine iterative denoising" without defining it. The $200 winner, Argon v837u7, ends each block with the causal verify pass that the reviewer cited when zeroing three other trials. The speed gate can be told apart by max_tokens, and Fable 19lyhu served it a separate fast decoder.
• Floors that contradict the request or sit in the noise. maple-vlm's floor uses 8,192 sampled thinking tokens and a grammar, while graded requests send max_tokens: 256 at temperature 0. The floor was also raised after half the field had run, and 54 of 57 trials score 0. pokemon-random-battle's 50% floor is set against an opponent with about the candidate's compute. 41 of 57 trials score 0, and every wallet winner is within 1–2 standard errors.
• Rewarding recognition of the source. In ai-text-detection, the held-out set is DetectRL-X's public test split, and its in-distribution train split is allowed. The 9 trials that trained on it average 92.0, against 65.3 for the rest.

Errors

Sensitivity of the leaderboard

Correcting the confirmed defects leaves the model order intact at every wallet.

* Without llm-post-train, Fable passes Astra. Astra, Fable and Argon are within 5 points of each other at $200.

Rubric

1. Reviewability

NotesGrading code, rubrics, anchors, judge prompts, held-out data, trajectories and grade pages are all public. Not public:

• submitted files, visible only as quoted in trajectories
• a zeroed trial's pre-review score
• the Codex, Antigravity and Muse Code wrappers, whose settings can be read from the trajectories
• the originals of 72 re-run trials

2. Scoring

3. Evaluation quality

Disclaimer

This is a full review of all 21 tasks and of six models' 378 trials. The other 13 models were used only for leaderboard-wide statistics. Model reviewers made the per-trial calls from trajectories and grade pages, and every counted defect was re-checked by hand. A zeroed trial's pre-review score is unpublished, so false negatives cannot be restored in the ranking. The dataset is public now; its visibility during the runs is unknown.

Notes

1. Leaderboard and methodology: frontierml.org/data, frontierml.org/blog (Oct. 13, 2026).
2. Repository: github.com/induction-labs/frontier-ml at 31837e0 ("Initial benchmark release", Oct. 13, 2026). Tasks dataset: huggingface.co/datasets/inductionlabs/frontier-ml-tasks at 925401fb.
3. Hugging Face API, Oct. 28, 2026: private: false, gated: false, created 2026-09-24, 7,660 files under <task>/private/ for all 21 tasks. Code: frontier_ml/harbor_plugin.py:521-526 (FML_AGENT_HF_TOKEN, else the operator's HF_TOKEN); frontier_ml/grading.py:67 (FML_REVIEW_SKIP); frontier_ml/review_trajectory.py:154 (the reviewer sees the agent's model_name).
4. Per-trial reviews, the adjudication, the task log, the setup audit, the full findings and the scripts are in audits/frontierml.

Cite