0.8B local model with 2,000 labels beats Jev's frontier API on banking data
kalyan_kpl · x · 2026-10-20
A developer benchmarked a 14.3M-parameter decision engine plus an 0.8B reader (vegam) against Jev's live API on real banking77 data. Zero-shot, the local model mostly abstains instead of guessing. After fitting and calibrating on just 2,000 labeled rows — 15 minutes on a single free T4, no fine-tuning, nothing leaves the machine — it beat the frontier API on both accuracy and calibration: route accuracy 0.853 vs 0.756, route ECE 0.029 vs 0.100; dispute accuracy 0.892 vs 0.846. Takeaway: frontier models generalize, calibrated local models adapt — with full data privacy. Reproducible via pip install vegaml.
More from Models
- User slams Xiaomi MiMo over hard-to-cancel $60 subscription — bookwormengr · 2026-10-20
- Smaug Open-Sources SOTA AI Image Detector, Hits 86.7% Recall at 5% False Alarms — bindureddy · 2026-10-20
- Anthropic Email Reveals Claude Mythos 5.1, Opus 5.5, Sonnet 5.5 in Expanded Cyber Verification Program — Sauers_ · 2026-10-20
- Dream-1 launches: 41% of subtasks in 5 minutes, outpacing top LLM agents — gharik · 2026-10-20
- Xiaomi's MiMo-V2.6-Pro is 'the first LLM that thinks like a teenager' — and it's actually good — hrishioa · 2026-10-20
- Cloudflare ships a new 'extraction model': unstructured data in, fields with probabilities out — ritakozlov · 2026-10-20