Kev 4B matches Jev on accuracy but Jev bills ~257 extra tokens per request, up to 12x cost
facethef · reddit · 2026-09-26
The Opper team benchmarked Kev 4B, Jared Palmer's Apache-2.0 fine-tune of Qwen3.5-4B, against Jev side by side on the same endpoint, using a fresh set of 362 items published after both models shipped (new arXiv papers, Stack Exchange questions, GitHub issues) to avoid data contamination.
Findings
- Accuracy lands within 2 points on every task — inside the noise at this sample size
- Jev is better calibrated and wins on paraphrase detection (PAWS 87.0% vs 74.5%)
- Same list price, but Jev counts a fixed 257 extra input tokens per request (also when calling TypeSafe directly), making short requests cost up to 12x more
Benchmark code, test items and results are open-sourced on GitHub. The takeaway: an open 4B fine-tune already matches the commercial model on capability, while billing quirks may matter more than accuracy for real-world cost.
More from Models
- Micius-Flash: 27B Multimodal Model for Hardware and Semiconductor Agents — zhiman-ai · 2026-11-11
- Claude IP risk-check site goes viral as users audit their IPs ahead of new rules — vista8 · 2026-11-11
- Claude IP purity-check site goes viral the day before new rules kick in — vista8 · 2026-11-11
- Baidu says RSI auto-optimized ERNIE inference, 6.24x decode speedup on Kunxin P800 — xiaohu · 2026-11-11
- Baidu's ECHO: multi-token prediction reusing the last layer as a drafter — xiaohu · 2026-11-11
- ERNIE 5.5 Flash Preview: 280B-param multi-router MoE activating only 14B per token — xiaohu · 2026-11-11