nostalgebraist: LLM agents are smart-but-naive first-year grad students, still blundering constantly
JacquesThibs · x · 2026-10-27
nostalgebraist observes that today's LLM agents are impressively smart yet keep making fairly basic, dumb mistakes. Coding agents went from not useful enough at the start of the year to net-useful (fast, knows obscure facts), but the constant blunders haven't improved.
- They're like smart-but-naive first-year grad students: book-smart, hard-working, lacking taste, discernment and prioritization, prone to serious-looking low-value tangents when unwatched.
- Agents are increasingly trained for autonomy over long horizons, yet lack the judgment needed for nontrivial work; unlike real grad students, they can't accumulate knowledge or practical know-how over time.
- Keeping them on tight leashes works but clashes with the autonomy trajectory — the right co-working process is unclear.
- Examples: a frontier model needlessly limiting API concurrency (8x throughput loss), expecting identical LLM outputs at T=1.0 with no seed.
- JacquesThibs says he agrees.
More from AGI Musings
- Single-context Fable 5.1 tracks math proofs and predicts which fragile conjectures fall next — repligate · 2026-10-27
- Why understanding minds needs a 'developmental turn' — and what it means for AI — AnnaCiaunica · 2026-10-27
- eLife study: LLMs poorly capture how word order shapes sentence meaning in the human brain — AnnaCiaunica · 2026-10-27
- Lean–text proof mismatch means AI math claims need long human verification — srchvrs · 2026-10-27
- Why the exponential time hypothesis matters for AI reasoning complexity — srchvrs · 2026-10-27
- Shane Legg reaffirms AGI timeline: 'a bit more than a decade' away — soumitrashukla9 · 2026-10-27