Transformers Are Samplers: Retry Harnesses Add Overhead and Formalisation Won't Fix Failure Rates
gerardsans · x · 2026-10-23
A technical debate on the limits of LLM code generation.
- The original post argues current LLMs are built on open-ended natural language, yet must handle both Shakespeare and Python; programming is rule-based with strict syntax and logic, but the architecture never adapted.
- The counterpoint: Transformers are samplers — generate N samples, k fail, and retry harnesses add considerable overhead in code generation. The loop is sample → verify → repeat, and formalisation keeps the exact same failure rate.
- Core claim: the failure rate on strict, rule-based programming tasks is structural, not something engineering wrappers can fix.
Related event: Debate: Transformers are samplers, limiting code reliability(3 posts)→
More from AGI Musings
- Montreal.AI paper: rejection-only experience transfer cuts candidate checks from 233 to 118 — Ghost_Pilot_MD · 2026-10-23
- Anthropic co-founder Jack Clark: Chinese models are 'too dumb', 6-12 months behind on safety — pstAsiatech · 2026-10-23
- Skip the thesis: strong researchers should publish and share, not write for nobody — gabriberton · 2026-10-23
- Swapping Claude for Gemini mid-run at step 433: disk-based init-d state files make agent handoff seamless — RandalSchwartz · 2026-10-23
- Stripe's Link adds purchase protection as AI agent "fat-finger" insurance debate heats up — jeff_weinstein · 2026-10-23
- Security researcher defends Anthropic: Jason Clinton warned of AI hacking influx 2+ years ago — JeffLadish · 2026-10-23