4,165 Agent Turns Later: Small Open-Weight Models Struggle at Real Coding Work
Vaibhav Jain · hf · 2026-10-27
Independent researcher Vaibhav Jain publishes an empirical retrospective (July–October 2026) on whether small open-weight models can do real coding work.
- Key finding: generating text is not the same as finishing coding work. Nominally compatible endpoints differ widely in tool-call formatting, turn control, context behavior, cache reuse, and reliability.
- "Served model," "coherent text," "executed tool calls," and "code passing independent behavioral checks" proved to be distinct levels of progress.
- Evidence includes 4,165 coding-agent gateway turns from his own sessions and 623 debug jobs across vLLM, SGLang, llama.cpp, and MLX serving setups.
More from coding & agent
- Cheap verifier matches frontier LLM on RAG claim checks at 187x lower cost, but passes 23.2% of unsupported claims — rohanpaul_ai · 2026-10-27
- From 5-Word Prompt to Physical Product: AI Designs and 3D-Prints a Filter Holder — blixt · 2026-10-27
- "The agent understood me" isn't a product requirement: put implicit decisions in the spec — shoumikgoswami · 2026-10-27
- Claude Code + Codex re-tested 8 famous research findings overnight — only 1 held up — beglen · 2026-10-27
- One word 'fricative' unlocked full IPA planning: why domain knowledge still wins with coding agents — mattpocockuk · 2026-10-27
- From 126 review comments to zero: letting agents auto-merge PRs — calabi_and_yau · 2026-10-27