Gergely Orosz: 3 AI Models Confidently Gave Wrong Legal Advice, the Lawyer Was Right
teropa · x · 2026-10-19
Gergely Orosz described an observation that messes with his mind: in areas where he's an expert (e.g., coding on projects he knows well), he immediately spots AI errors and hallucinations; but in unfamiliar domains he tends to trust the model — and later finds it did stupid things.
His concrete case: he asked 3 different AI models for feedback on a legal matter, and all three confidently said X. His (slower) paid lawyer said Y — and the lawyer was right. Quoting vladmihalcea: Claude Opus 5.5 often matches how he'd implement code himself, but occasionally makes terrible design decisions that would fail in production — a reminder that software engineering will remain valuable for years.
More from AGI Musings
- 'Math is infinite': AI proof drama sparks backlash against whining mathematicians — MoonL88537 · 2026-10-19
- Pre-ICLR bet: researchers wager 90% of AI fellowship papers will be rejected — anpaure · 2026-10-19
- Snarky startup idea: LLM-faked 'leaked emails' where redaction bars do the narrative work — l4rz · 2026-10-19
- Princeton's Arvind Narayanan compiles ~100 posts on research careers and AI's impact on work — random_walker · 2026-10-19
- TAIS 2026 safety panel: longer chain-of-thought makes models easier to jailbreak — GaryMarcus · 2026-10-19
- "Free will is an illusion under superintelligence" — blogger invites debate — VraserX · 2026-10-19