Harness skeptic: model training-data support drives performance, not the harness
gerardsans · x · 2026-11-15
Replying to the call for public harness benchmarks, @gerardsans calls it a common misconception: what drives performance is the model's training-data sampling support for a task, with the harness only secondary. "Inference needs distribution mass. There's no way around it," he writes, citing numerous controlled experiments in the literature.
More from coding & agent
- TensorFold PR fixes MCP servers blocked by macOS sandbox over dyld framework loading — AIFlow_ML · 2026-11-15
- Alexey Grigorev: agent instructions shouldn't describe their tools — Al_Grigor · 2026-11-15
- Engineer with unlimited local DeepSeek V4.1 Flash tokens asks what's actually worth running — AbbreviationsLoud182 · 2026-11-15
- "Phonebook-trained models can't write Shakespeare": data distribution beats any harness — gerardsans · 2026-11-15
- 2 hours and 2M tokens: Fable 5.2 builds a stunning procedural 3D game — FinanceYF5 · 2026-11-15
- 20 skills and open-source tools that turn Fable 5.2 into a full video production pipeline — FinanceYF5 · 2026-11-15