LALA Bench: 236 Academic Questions Expose LLMs Answering from Memory, Not Reading
JasonBotterill · x · 2026-10-28
JasonBotterill introduces LALA bench — 236 open-ended questions on linguistics, logic and literature, written by academics, testing how well models actually analyze sentences and texts. The recurring failure: models offer familiar interpretations without attending to the prompt's wording, overlooking tense changes, missing alterations to well-known literary passages, or choosing a single referent where several are possible. LALA's deliberate distortions (intentional typos, confusing phrasing) probe how easily models are thrown off by detail; a good answer must recover the intended meaning and flag remaining ambiguity.
More from Research
- Real human laughter remains AGI-hard, argue AI improv researchers — korymath · 2026-10-28
- Johns Hopkins lands up to $52M from ARPA-H to build AI that decodes protein misfolding — JosephJacks_ · 2026-10-28
- Microsoft's Diagnostic AI published in Science and open-sourced, retested on Gemini 4 Argon — vivnat · 2026-10-28
- Machenta robot reacts and recovers in real time, trained 100% in simulation — Ed__Johns · 2026-10-28
- DeFM Depth-First Foundation Model: Why Anchor to Depth, Not Estimator Features — chris_j_paxton · 2026-10-28
- AI-powered formal verification finds real Redis/Valkey bugs, incl. a 23-char freeze — LingmingZhang · 2026-10-28