Small model Jev matches Opus on 95% of few-shot classifier calls; disagreements refine prompt
honnibal · x · 2026-10-21
Matthew Honnibal (spaCy creator) shares a hands-on case: a user project relied on many few-shot classifiers powered by Opus. Running Typesafe's Jev over the same data, Jev agreed with Opus on 95% of inputs.
He then queued the disagreeing samples for human annotation, which pinpointed a few necessary prompt refinements for Jev (screenshot shows a three-way NEGATIVE/NOTNEGATIVE/UNCLEAR annotation UI).
It's a reusable workflow: swap an expensive frontier model for a small one on classification, then iterate the prompt via disagreement-driven annotation.
Related event: spaCy author finds small Jev model matches Opus classifier 95% of the time(2 posts)→
More from coding & agent
- 90% of big-company knowledge lives in heads, and "company brain" startups are in for pain — gabriel1 · 2026-10-21
- EAU: Co-evolving user simulator, environment, and RL setup for training LLM agents — UAlberta · 2026-10-21
- Meta's Self-Credit: Dense Rewards from Sparse RLVR via Intra-group Rescaling — meta · 2026-10-21
- Notion details its Auto model router: Elo-based evals, 62-93% cheaper than flagship models — ivanhzhao · 2026-10-21
- Dev sets up automation so agents proactively pick up stale issues and PRs — vikvang1 · 2026-10-21
- Tamper-proof agent logs still can't prove the agent did the job: who reads the record back? — ZealousidealAct9661 · 2026-10-21