Open-source 450M decision model Nirnay beats Jev on Banking77, runs on CPU in 361ms
BrilliantSecret143 · reddit · 2026-10-02
Eulogik released Nirnay, an open-source (Apache-2.0) decision model for intent classification and routing: 450M params (a Laya fork plus 30M), one forward pass yielding calibrated probabilities with no text generation.
- Results: 0.8792 accuracy on Banking77 (3,080 cases, Brier 0.208, fitted ECE 0.045), vs 0.803 for Jev 1.13.0 zero-shot. The author openly flags the unfair comparison — Nirnay was fine-tuned — framing the thesis that fine-tuning a small model beats API calls on your own data.
- Speed: 209ms on M4 GPU, 361ms on CPU (batch-1, PyTorch). No GGUF/Ollama build yet; a Python env is required.
- Also in the repo: write-ups of two training collapses and their fixes (150x scale runaway, silent usage collapse to 1/77 classes), a 9-page paper draft, and all evals as raw JSON. JevBench-hard is weak (0.396 on long docs) and published as-is.
Weights on HuggingFace (eulogik/nirnay-450m), pip-installable.
More from Models
- AI's clever quantization move is just sensitivity analysis, an old pruning trick — code_star · 2026-11-16
- AI progress doubled since GTA VI's first trailer; GPT-4 Turbo now ranks 379th — ArtificialAnlys · 2026-11-16
- Tencent's Hy4 Flash builds a 3D game with shield mechanics in 63 minutes, beating Hy4's 90 — ShunyuYao12 · 2026-11-16
- Hy4 Flash 3D voxel demo outpaces DeepSeek-V4.1 in same-prompt comparison — ShunyuYao12 · 2026-11-16
- Yacine MTB claims training LLMs is 'easy', says he's shocked by how simple it is — yacineMTB · 2026-11-16
- MiniMax M3.1 Flash lands on OpenRouter, with Inferact as first provider — Xianbao_QIAN · 2026-11-16