"It Was Never the Model": 11 Falsehoods in a Local Inference Stack, All From Tooling
colinmcnamara · x · 2026-10-13
Engineer Colin McNamara built a local inference setup—a laptop, a single GH200, and a semantic router meant to send trivial turns to a small local model and hard ones to a 27B—and found the premise didn't survive measurement. Eleven confident falsehoods in his stack, and not one came from the model; three came from his own instrumentation, including cache-busting logic that was itself cached, turning a 24x gap into apparent parity.
The article details the architecture: two client dialects (Anthropic messages API, OpenAI chat) feed a router reading four signals and making three decisions; one GH200 runs the 27B, a speech model, and a guard model in 78 of 95.6 GB HBM3, with 573 GB LPDDR5X on the Grace CPU joined via NVLink-C2C at 447 GB/s. He also honestly flags that speech bypasses the router and the guard model is measured but not wired.
Related event: Engineer Finds 11 Confident Errors All Came From His Toolchain, Not the LLM(2 posts)→
More from coding & agent
- Eleven: free open-source tool lets Claude, Codex and Cursor agents do binary reverse engineering — emeka_boris · 2026-10-13
- How Every cut its AI agent's token costs: caching, less reading, Opus 5.5 — every · 2026-10-13
- 81% of engineering leaders hit AI-related production incidents in six months — sanjaykalra · 2026-10-13
- ShipStation's Slack IT agent on Bedrock AgentCore resolves 97.5% of queries — AWS ML Blog · 2026-10-13
- Agent Buddy: an open-source ESP32 desk device that watches your coding agents — DanWahlin · 2026-10-13
- AWS benchmarks Grok 4.7 agents across 72 runs: xhigh is perfect, low uses 1/8 the tokens — AWS ML Blog · 2026-10-13