Minor Harness Setting Tweaks Radically Alter ARC-AGI Scores
teortaxesTex · x · 2026-08-01
Recent discussions around the ARC-AGI-3 benchmark reveal a crucial observation: minor changes to the execution framework's settings can significantly impact a model's final score.
Developers noted that simply making two adjustments to the Harness drastically improved Claude 3.5 Sonnet's performance. This raises concerns about the accuracy of current LLM evaluations and suggests that the true capabilities of models like DeepSeek V4 might be underestimated by existing testing frameworks.
Related event: ARC-AGI 3 Evaluation Mechanism Under Fire from Developers(9 posts)→
More from coding & agent
- 'LLM code is sloppy' rebutted with real ffmpeg code: identical if/else branches in a top library — kingslyj · 2026-10-16
- Agent actions need checks outside the model: NVIDIA demos lean on guardrails amid wave of real agent incidents — Creamy-And-Crowded · 2026-10-16
- Vitalik builds "paranoid mode" agent wrapper with Tor, zkAPI and local prompt rewriting — DavideCrapis · 2026-10-16
- Agent fine-tunes Qwen3.5-0.8B with synthetic data: 4.7%→76.3% accuracy, 11.9× faster — ben_burtenshaw · 2026-10-16
- Beam-501B agent demo: solo beats 5-agent team on ProgramBench rebuild task — ben_burtenshaw · 2026-10-16
- Beam agent builds BeamSpeak, a local Parakeet dictation app for macOS — ben_burtenshaw · 2026-10-16