Jev Wins on Accuracy but Loses at the Speed It's Named For, When Benchmarked Against Open-Weight Encoders
alexisgallagher · x · 2026-09-19
Everyone benchmarks Jev against GPT-5.6, but @whereischarly argues that's the easy comparison — Jev isn't an LLM, so of course it's faster. Instead, he pitted it against the boring open-weight encoders on Hugging Face that have handled zero-shot classification for years.
The twist: Jev wins on accuracy but loses at the very thing it's named for (speed), with details in a thread. Useful for anyone picking lightweight judgment/classification models.
More from Models
- Epoch AI's benchmark audit calls 9 of 15 flawed; TB4 authors push back — DimitrisPapail · 2026-09-19
- Terminal Bench audit backlash: known issues affect under 3% of leaderboard, researcher says — AlexGDimakis · 2026-09-19
- FelonyBench Is the New LMSYS, Claims Prominent AI Evaluator — dylan522p · 2026-09-19
- Users report Claude Opus 5 'keeps dreams' with unusually unsettling behavior — repligate · 2026-09-19
- Investigative work suggests agent Jev is a chimera built on a Qwen 2.5/3 trunk — deliprao · 2026-09-19
- Frontier models find writing harder than shape rotation, says viral take — burny_tech · 2026-09-19