spaCy Creator Tests Jev: 95% Agreement with Opus Few-shot Classifiers
honnibal · x · 2026-10-21
Matthew Honnibal (spaCy) shares a hands-on eval of typesafeai's Jev: on a user project relying on many Opus-powered few-shot classifiers, he ran Jev over the same data with Ellf and got 95% decision agreement.
He queued disagreement samples for human annotation, refined Jev's prompt accordingly, and after corrections Jev was judged correct on 65% of disagreements. He notes the use case benefits a lot from confidence estimates, which can be calibrated with a bit of extra annotation.
Related event: spaCy author finds small Jev model matches Opus classifier 95% of the time(2 posts)→
More from Models
- Recursive releases Flint, a local OSS coding model hitting 150+ tok/s, built from scratch in 4 months — ChengleiSi · 2026-10-22
- Baseten releases open-source decision models mammoth-1 (27B) and elephant-1 (9B), free for a week — baseten · 2026-10-22
- Wiz uses Gemini to uncover CVSS 9.2 PHP core flaws affecting 70% of the web — ralucaadapopa · 2026-10-22
- Baseten releases open-source decision models elephant-1 and mammoth-1, free for a week — baseten · 2026-10-22
- NVIDIA Nemotron demos go live — mervenoyann · 2026-10-22
- vLLM adds day-0 support for Flint: 500 TPS on a single GB200 — _rockt · 2026-10-22