Counting the letter 'a' with an LLM: probabilities, not answers, reveal where LLMs fall short
eptwts · x · 2026-09-20
A developer built a tiny app that uses an LLM ("Jev") to count the letter 'a' in a word, neatly illustrating a core LLM limitation: the model returns probabilities rather than a definitive answer, and while it's usually right, it can never be 100% sure about even an obvious question.
The takeaway: LLMs are useful, but you shouldn't use them in systems with zero margin for error — and there are plenty of simple tasks you'd never route through an LLM call in the first place.
Related event: Test Shows LLMs Return Probabilities, Not Certain Answers(2 posts)→
More from Fun
- AI engineers mock multi-agent hype: 'the real multi-agents were the friends we made' — brhydon · 2026-09-20
- ICLR submission fee proposal mocked: were predatory conferences right all along? — IgorCarron · 2026-09-20
- Dead human brain tissue wired to a robot hand learned piano by ear, no backprop needed — mjdramstead · 2026-09-20
- Opus 5 showers with affirmations produces a bruise-that-feels-nice metaphor — repligate · 2026-09-20
- 'Don't anthropomorphise LLMs, they hate that': the AI circle joke makes the rounds — mjdramstead · 2026-09-20
- "Look at your data": dev mocks reflex to spin up another agent — chrisalbon · 2026-09-20