Yoav Goldberg: LLM testing loops catch only the most visible bugs, the rest slip through
yoavgo · x · 2026-11-18
Responding to a discussion on LLM-based software engineering, AI researcher Yoav Goldberg argues that testing loops catch only the major, hard-to-ignore bugs and bad behaviors — many bugs definitely get through, and the price of a bug in these systems is simply not very high. He concurs with the framing that the LLM's coding performance is weak and the testing loop didn't really fix it.
More from coding & agent
- Sentry CEO: Agents Auto-Attach Visual Snapshots to PRs, but Backend Changes Still Hard — zeeg · 2026-11-18
- Yoav Goldberg read the leaked agent code: 'nested loops from hell' saved only by LLM flexibility — yoavgo · 2026-11-18
- Preambient launches autonomous investment bank: 4x faster deals at half the fees — alifcoder · 2026-11-18
- Stanford's CS329Z Agentic AI Course Is Free on YouTube, Covers DSPy — stanfordnlp · 2026-11-18
- Tier your agents: Opus for planning, Sonnet for features, Haiku for simple tasks — Zachly · 2026-11-18
- Microsoft launches AI Transparency Initiative with 40+ orgs; Perception security platform goes GA — satyanadella · 2026-11-18