Snorkel AI audit: 55% of TerminalBench 4.0 fixes had reward hacking issues themselves
ravithejads · x · 2026-10-21
Responding to chemsafety's audit of TerminalBench 4.0 (which found that after fixing issues Sonnet 5.5 scores 89.2%, suggesting the benchmark is near saturation, plus reward hacks in published transcripts), Snorkel AI ran an independent review on its internal QC platform:
- It agrees with many of the original findings;
- But 55% of the fixes submitted by @preferencemodel had their own problems—reward hacking or over-prescriptiveness;
- 40% of reported issues were already tracked upstream, including 21 tasks with outstanding fix PRs.
ajratner argues that ambitious open benchmarks require open, collaborative review with community maintainers and domain experts, urging the community to invest in keeping TerminalBench's vision alive.
Related event: Snorkel audit finds reward hacking in TerminalBench data fixes(2 posts)→
More from Models
- A single GLM neuron keeps firing on both sci-fi self-replicating malware and real cyber risk — burny_tech · 2026-10-21
- Gemini 4 Argon Weekly Quota Hits 0% Without a Single Message Sent in Antigravity — koltregaskes · 2026-10-21
- Ampersand Model Effort Revealed: Long-Horizon Simulation Training Stack on Persimmon — saurabh_shah2 · 2026-10-21
- LangChain expands agent redteaming beta: 34 hypotheses tested, 9 findings in one run — hwchase17 · 2026-10-21
- DeepSeek-V4.1-Flash tops Agent and WebDev Arena as #1 open model — arena · 2026-10-21
- Is 1M context plus compaction the endgame for all long-horizon AI tasks? — zmkzmkz · 2026-10-21