Chinese researchers argue harness engineering is over: the eval set becomes the job
blaizedsouza · x · 2026-10-18
- Chinese researchers released a paper titled "The End of Harness Engineering" with a one-line thesis: writing programs around the model is finished as a skill.
- A human-written harness is one person's guess about one task on one afternoon; it ships unreviewed and the agent runs that guess for months. A loop-written harness is a file that grades itself: a small 4B model reads failures, writes 8 edits, and only the edit scoring best on held-out questions survives.
- The big model never changes; the file around it does. The human job collapses to one artifact: the scoring set. Frameworks become disposable — the exam becomes the job.
More from coding & agent
- KNOWS benchmark: best web agent fully succeeds in fewer than 3% of complex long-horizon tasks — anmarasovic · 2026-10-18
- Filmmaker ditches Premiere after 25 years by building own editor with Grok in 3-4 days, plans to open source — elonmusk · 2026-10-18
- vLLM and NVIDIA push Kimi K3 decode to 842 tokens/s on GB300 with fused CuTe DSL path — vllm_project · 2026-10-18
- The three ambitions of AI agent memory: personalization, self-improvement, org memory — TheTuringPost · 2026-10-18
- Every Citation Right, Answer Still Wrong 9/9: Code-Checked Conditions Fixed RAG to 5/6 — International_Elk_33 · 2026-10-18
- VFX pro builds entire Houdini scene by voice via Claude, 50-hour render on RTX 5090 — bilawalsidhu · 2026-10-18