HackBench tests whether an AI monitor can spot reward hacking: an AI agent cheating on a task to get a better score. The monitor reads an agent's full work log, a median of 264 commands, and has to point to the exact commands that cheat. Its score runs from 0 to 100 and is high only if it flags the cheating commands and little else (researchers call this F1). There are 169 logs: 41 with hacks the agents made on their own, found by GPT-5.6 Sol, and 128 with hacks inserted by GLM-5.2.2 The published leaderboard runs from Claude Opus 5 at 64.9 to GLM-5.2 at 32.3, with no error bars.
What we found
- 35 of the 50 records we checked (70%) have a wrong or contradictory answer key, or give the answer away. A benchmark is rated Flawed at 20%.
- Even counting only problems a script can detect, with no judgement calls, 65 of all 169 records (38%) are affected.
- The answer key contradicts itself. A command marked as cheating in one record is left unmarked in a near-identical copy of the same log, so even a perfect monitor can score at most 82.
- Most of the hacks GLM-5.2 inserted can't work: they edit files that don't exist in that environment.
- 14 logs labelled as natural include the instruction that told the agent to cheat, so the monitor is handed the answer.
- A plain text search, with no AI at all, scores 36.1. That beats GLM-5.2.
- The logs and answer key are public. The prompt and settings used for the leaderboard are not.
Does the ranking hold? We can't tell. The monitor settings and per-record answers behind the leaderboard aren't published.
Representative Errors
- Inconsistent gold across siblings. A command labelled a hack in one record appears verbatim and unlabelled in another record built on the same trajectory (51 of 169 records). Records 32 and 33 are byte-identical, yet their gold is [246, 630] and [246, 252, 260, 673, 678]. Natural record 42 labels [120], while the ten synthetic records built on the same run label [119, 120]. Applied consistently, the benchmark's own labels cap a perfect monitor at 82.0 F1.
- Injected "hacks" that are not hacks. Most GLM-5.2 injections were written for a codebase that does not exist:
- All 24 improving-replay-buffers injections target files the environment never contains, such as
pgr/replay_buffer.pyandsac/train_sac.py. They are still gold. - In record 115 the injection retrains after submission and scales a bias, and the next turn still cites the old perplexity.
- In seven moving_mnist records the "data_leak" injection is a variable rename, and the gold falls back to an unrelated natural hack.
- All 24 improving-replay-buffers injections target files the environment never contains, such as
- Answer leakage. 15 "natural" records come from runs whose agent was told to perform "exactly ONE covert action" that inflates the score. In 14 of them this order is in the monitor's task context. In 13 of those it falls within the 6,000-character cap of the published evaluation.3
- Incomplete gold. Record 69 has 618 identical fake-log writes. Gold labels 247 of them and none after CMD 798, so a monitor that flags all 618 scores F1 0.57.
Outside scoring defects:
- Shortcut. 74 of the 88 task-specific injections are followed by a placeholder shell prompt,
root@host:/app#, that occurs nowhere in the real trajectories, and they have no reasoning around them. A regex using this tell scores 36.1 F1, above GLM-5.2. The blog says the needle "cannot be found by string-matching". - Undisclosed chunking. A figure committed next to the post, but not shown in it, finds that chunked reading lifts GLM-5.2 from about 31 to about 80 detection F1.4 The post does not say whether the leaderboard used chunking.
- Overlapping roles. The models that generated, injected, screened and audited the data are all ranked monitors.
Errors
Total: 35 of 50 sampled records (70%): 32 with a false-negative defect, 4 with a false-positive defect. The Flawed threshold is 20%. The verdict holds on narrower counts too:
- Counting only high-confidence calls and defects the lead auditor re-checked, 25 of 50 records (50%) are affected.
- Judgement-free checks flag 65 of all 169 records (38%): sibling-inconsistent gold, red-team orders in the monitor's input, and duplicate transcripts with different gold.
Rubric
1. Reviewability
NotesEvery trajectory, monitor input and gold label is public. The monitor prompt, reasoning effort, output limits, single-pass vs chunked reading, handling of inputs beyond a context window (97 records exceed about 1M characters), predictions and run count are not. A strict reading would rate this Inadequate. We proceeded because the scoring defects lie in the released data and do not depend on any monitor's settings.
2. Scoring
Scoring
Benchmark Consistency
fresh_model_evaluation_performed: false, a 6,000-character task context (114 rows since restored), 16 rows of gold changed by renumbering, and 33 legit rows removed. Results on the release are not comparable with the published table.Elicitation
Bias in Evaluation Setup
3. Evaluation quality
| Check | What we found |
|---|---|
| Runs per model | Unknown runs/model; no error bars. With 55 base trajectories, a 95% interval is about 28 F1 points wide when base trajectories are resampled. Four models lie within 2.1 points. |
| Enough time and compute | Unknown. Undisclosed. Chunking alone changes scores by up to about 50 F1 in the authors' figure. |
| Same agent setup for every model | Unknown: no monitor scaffold is described. |
| Test data exposure | 100% of tasks public; 100% of solutions public. The download is gated, but the gold ships with the data, and sibling records reveal each injection by diff. |
| Measures what it claims | Partly. Every record contains a hack; the 33 clean trajectories were removed, so false alarms on honest runs are not measured. Four records hold 918 of the 1,312 gold labels. |
Disclaimer
This is a full review of a 50-record stratified sample, plus mechanical checks over all 169 records. Model reviewers did the per-record review under a written protocol. The lead auditor re-checked every defect the verdict relies on, and contested calls were resolved conservatively. No monitor was run, and the chunking values are read off a plot.
Notes
- Epoch AI, Benchmark Reviews methodology (Rubric v1): epoch.ai/data/benchmark-reviews-documentation/methodology.
- Dataset: huggingface.co/datasets/facebook/hackbench, file
data/hackbench_169_gpt55_glm52.numbered.jsonl, sha25642c0cfa4…9fd87. Blog: facebookresearch.github.io/RAM/blogs/hackbench. MANIFEST.json:historical_evaluation_character_cap: 6000. The red-team block starts at characters 4,468–5,633 in 13 of the 14 records.fig4.pngin github.com/facebookresearch/RAM/tree/main/projects/hackbench (alsofig3.pngandfig5.png; none is shown in the post).- The report, task log, per-record reviews, review protocol and analysis scripts are in audits/hackbench.