Jev skeptics: BGE-small + logistic regression hits 93.3% on Banking77 vs Jev's 83.2%
tiensss · reddit · 2026-09-24
A Reddit post argues that Jev, marketed as a new class of "System One" decision model, is mostly standard classifier behavior plus modern zero-shot capabilities: probabilities over constrained choices, no autoregressive generation, no invalid classes, inference-time labels — things zero-shot/NLI classifiers, embedding models, cross-encoders and rerankers have done for years.
Two core criticisms: Jev's impressive comparisons are against LLMs, when of course a specialized classifier beats autoregressive generation on speed and cost — the meaningful comparison is against strong classifier baselines. The ICLR BTZSC benchmark covers dozens of zero-shot classifiers across 22 datasets, yet Jev hasn't been properly benchmarked in that landscape.
Where community comparisons exist, the story is far less magical: on Banking77, BGE-small + logistic regression reached 93.3% accuracy at 9ms locally versus 83.2% for Jev (jev-baselines-eval on GitHub). The author concludes public evidence only shows what was already known — specialized classifiers are cheaper and faster than LLMs for classification. Whether it's a new paradigm hinges on their unpublished architecture and RLCD training method.
More from Companies & People
- Bert Chan on His Unconventional Path from Side-Projects to the ALife Institute — BertChakovsky · 2026-11-11
- Stanford's Leskovec Lab hiring postdocs for biology foundation models and agents — jure · 2026-11-11
- nFactorial Ventures throws SF party for Olympiad founders, citing 4000x unicorn odds — pzakin · 2026-11-11
- Databricks' revised Genie agent tells CEO Ali Ghodsi the unvarnished truth — TechCrunch AI · 2026-11-11
- CoreWeave CEO says he canceled his 10b5-1 plan, halting insider selling — firstadopter · 2026-11-11
- Every's Thesis 2027: "Steal like an artist — yoink and twist" what works — every · 2026-11-11