Terminal-Bench thanks community auditors, publishes task flaw reporting process
ajratner · x · 2026-11-11
The Terminal-Bench team thanked Bespoke Labs, Preference Model, Epoch AI, Tokenless and Snorkel AI for auditing Terminal-Bench and sharing results with the community, encouraging deeper collaboration on severity assessment and merging high-quality fixes.
They also published a process for reporting new task flaws: manually verify the flaw, assess downstream impact, open a concise GitHub issue and tag task authors; motivated contributors can present findings at the weekly meeting, co-work on fixes, and add checks to automated task QA tooling.
More from Research
- ProtoMotions Deployment Stack: One Exported Graph, 1.3 ms C++ Loop for Real Robots — carlosdponx · 2026-11-11
- DeepMind's MKA benchmark splits frontier agents: 95-97% on basics, 28-57% on hard tasks — ArtificialAnlys · 2026-11-11
- MKA deep dive: top models cluster at 95-97% on basics but split 28-57% on hard tasks — ArtificialAnlys · 2026-11-11
- MKA Benchmark by Output Type: No Model Tops 52% on Word or Text Files — ArtificialAnlys · 2026-11-11
- DeepMind-backed MKA benchmark crowns Claude Fable 5.2 at 57%, with 30x cost spread — ArtificialAnlys · 2026-11-11
- Emergent misalignment is predictable OOD; later-layer fine-tuning reduces transfer — AdtRaghunathan · 2026-11-11