Testing Jev on counting letters: it returns probabilities and still flubs obvious questions
eptwts · x · 2026-09-20
A developer built a tiny app that uses Jev to count the letter 'a' in a word, exposing a key limitation:
- The model doesn't return a definite answer — it returns probabilities and is rarely 100% sure.
- It gets most cases right but still flubs blatantly obvious questions.
- Takeaway: the model is useful, but shouldn't power zero-margin-of-error systems; the sweet spot is fast, structured judgments over messy information.
Related event: Test Shows LLMs Return Probabilities, Not Certain Answers(2 posts)→
More from Models
- Jev device review: fast and cheap, but C- at generalization, needs GPT-3-era prompting tricks — intellectronica · 2026-09-20
- Bespoke Labs open-sources Nimble, an open Jev rival for fast typed decisions — RexDouglass · 2026-09-20
- LFM2.5-230M hits 192 tok/s on a phone CPU in llama.cpp test — Tall_Abrocoma_3533 · 2026-09-20
- Researchers Pan Apple Foundation Models CLI Chat as 'Embarrassing' — StefanoGogioso · 2026-09-20
- ChatGPT-6 Astra Cracks 108-Year-Old Unsolved WWI German Code — Andune88 · 2026-09-20
- Using Jev, the 'System One' classifier, to speed up self-hosted agent memory — FUNdationOne · 2026-09-20