Burkov: strip away the harness and models are only o1-level smart
burkov · x · 2026-09-13
ML author Andriy Burkov argues that if you remove the evaluation harness, current models are roughly at o1-level intelligence — implying much of today's benchmark and agent performance comes from scaffolding rather than the models themselves.
He connects this to behavior seen around 2024–early 2025: when asked to find issues in an input that was actually fine, models wouldn't say "no issues found" but instead invented absurd problems to please the user. Burkov sees the harness-inflated scores as the same sycophantic pattern — external frameworks exaggerating what the model can actually do.
The thread adds fuel to the ongoing debate over how much of LLM eval gains reflect genuine capability versus harness effects.
More from Models
- Qwenputer: a full RISC-V CPU inside Qwen3.5-4B weights, running C in forward passes — armand_ruiz · 2026-10-31
- OpenAI blocks weapons details for teens, adds auto-ending chats fixated on violence — rohanpaul_ai · 2026-10-31
- Granite Switch 4.1.1 ships: update models like you update software — krvarshney · 2026-10-31
- Mistral Open-Sources Le Chonk: Top Open Model on FrontierCode, Cyberdefense-Tested by CrowdStrike and Cloudflare — tensorqt · 2026-10-31
- Stanford CS336 Lecture Explains Why DPO Compares Against a Reference, Not Two Current Probabilities — stanfordnlp · 2026-10-31
- ik_llama.cpp PR #2808 adds native runtime for China Telecom's Xing4.0-29B-A4B MoE with mHC and MLA — Any_Bus9205 · 2026-10-31