Models and Harnesses Are Co-Designed Systems, Not Engine Plus Wrapper — Same Model Scores Differently on SWE-bench
GaryMarcus · x · 2026-10-12
A post retweeted by Gary Marcus argues that in coding, the model and harness are best understood as a co-designed system rather than an engine with a detachable wrapper: the same model scores very differently on SWE-bench depending on the harness.
It also defends symbolic verifiers (automated theorem provers) against being called "basic": they are the crowning achievement of 20th-century mathematical formalization. It took Frege's quantifier logic, Russell's types, the formalist program and Gödel's critique, Church's lambda calculus, Curry–Howard, Martin-Löf, and Coquand & Huet's Calculus of Constructions — all so that "is this proof correct?" could collapse into a compact, mechanical type-checking procedure a computer can run.
More from coding & agent
- DominikTornow: We need abstractions for specifying and verifying code, not just writing it — _AustinCalvert_ · 2026-10-12
- Forget "prompt": better to ask what happened that allowed THIS to happen — liminal_bardo · 2026-10-12
- From apps to RL environments to personal agents: the AI startup ladder keeps shifting — realsohamparekh · 2026-10-12
- Predicting a 75% chance Anthropic or OpenAI ships an LLM-optimized programming language this year — dh7net · 2026-10-12
- 'Software is dead' takes get the camera analogy rebuttal: it's software's Kodak Brownie moment — mgill25 · 2026-10-12
- CUT, a Timeline Editor for MiniMax H3 in ComfyUI, Now Free in Original Version — rileygstaliger · 2026-10-12