Open-source 450M decision model Nirnay beats Jev on Banking77, runs on CPU in 361ms
BrilliantSecret143 · reddit · 2026-10-02
Eulogik released Nirnay, an open-source (Apache-2.0) decision model for intent classification and routing: 450M params (a Laya fork plus 30M), one forward pass yielding calibrated probabilities with no text generation.
- Results: 0.8792 accuracy on Banking77 (3,080 cases, Brier 0.208, fitted ECE 0.045), vs 0.803 for Jev 1.13.0 zero-shot. The author openly flags the unfair comparison — Nirnay was fine-tuned — framing the thesis that fine-tuning a small model beats API calls on your own data.
- Speed: 209ms on M4 GPU, 361ms on CPU (batch-1, PyTorch). No GGUF/Ollama build yet; a Python env is required.
- Also in the repo: write-ups of two training collapses and their fixes (150x scale runaway, silent usage collapse to 1/77 classes), a 9-page paper draft, and all evals as raw JSON. JevBench-hard is weak (0.396 on long docs) and published as-is.
Weights on HuggingFace (eulogik/nirnay-450m), pip-installable.
More from Models
- JevBench to add evals for LLM routing, RAG retrieval, and moderation use cases — airesearch12 · 2026-10-02
- Google's Argon battle-tested by 200k+ Googlers daily, not benchmaxxed — Zergylord · 2026-10-02
- JEV claims to be the first System One model hosted in the EU — juanviera23 · 2026-10-02
- Gemini knew a user's mom's name unprompted, then gave three conflicting explanations — Exact_Firefighter864 · 2026-10-02
- GPT-6.1 Sol nearly matches Astra at a quarter of the price on RareBench — danielmckinn0n · 2026-10-02
- New results from PostTrainBench v1.2 are in — mariofilhoml · 2026-10-02