Agent ran same task in 40s vs 261s — the bug was his own requirement checker, not the model
Stunning-Sherbet1853 · reddit · 2026-11-02
A developer running a browser agent on a local 3-bit quantized 27B model (single RTX 3090) chased massive run-to-run latency swings (40s to 261s) on a bookstore search task.
Ruled out one by one:
- Thinking mode flips rebuilding the KV cache (one answer flip caused a 77k-token reread)
- History cap sized for a 131k window while running 192k, folding the transcript at 40%
- Page extractor picking an 18,000-char hidden category menu over the book list (fixed one bad run: 373s→124s)
Actual culprit: a small rule layer that converts requests into checkable requirements read "top 3 books: title, author, price" as "open 3 items and read each." The model already had the full answer by step 5, but the checker insisted the task wasn't done, forcing pointless page visits that looked exactly like a confused model in logs.
The fix: fields visible in a list (author, price, star counts) become "collect from the list." Regression-tested on all 2,810 logged requests — only 14 changed, all in the intended direction. The task now runs 44–64s consistently. Remaining unknown: whether any thinking setting helps, since run-to-run noise dwarfs the differences.
More from coding & agent
- DeepSeek Harness Super-agent design: the bottleneck isn't smarter models, it's your attention — sujingshen · 2026-11-02
- tito: the hidden tokenizer bugs breaking RL training, and a Lean-verified fix — VoidAsuka · 2026-11-02
- Build an Agentic Knowledge Graph from Scratch — No Graph DB, No Extraction Model — adnan_hashmi · 2026-11-02
- One Vault, Two Sides: An Obsidian + Claude Agent Workflow for Personal Knowledge — adnan_hashmi · 2026-11-02
- Vercel engineer open-sources Security Audit skill you trigger via @bot reply — brandon_galang · 2026-11-02
- Vercel Labs shipped 95 releases in October, debuts agent-desktop for computer use — evilrabbit_ · 2026-11-02