Fine-tuned Inkling 2 Mini routes tickets with early exit: median latency 136.8ms to 81.4ms at 0.07% accuracy cost
eltokh7 · reddit · 2026-11-23
A developer released Inkling 2 Mini Decision, an Apache 2.0 fine-tune that returns answer probabilities without generating tokens, letting a looped transformer spend fewer passes on easy decisions.
- A duplicate-charge ticket routed to billing at 98.7% after one loop; the escalation question ran all three loops and stayed near 50/50.
- On decision-v7/test, 54.2% of questions exited after one loop; median batch-one GPU latency on an RTX PRO 6000 dropped from 136.8ms to 81.4ms while accuracy moved just 85.56% to 85.49%.
- On the external Typed Decisions benchmark, 89.6% of questions still ran all three loops, showing early exit only pays off in-distribution.
Weights and eval scripts are open on Hugging Face.
More from Infra
- Splash 1.7.1 cuts Apple Silicon TTFT 23% via Neural Engine prefill, 2.9x faster shared-prompt chats — oss-jiiim · 2026-11-23
- From-scratch Metal 4 engine runs Inkling-2-Mini at 144 tok/s on M5 Ultra, 128k context in 32GB MacBook — DankpawsDev · 2026-11-23
- LFM3-Omni-2B runs locally on iPhone 17 Pro Max doing agentic web search with native speech generation — fuzhongkai · 2026-11-23
- Custom compiler runs Qwen3.5-0.8B BF16 on one CPU core with 109MB peak RSS via disk reads, seeking baselines — Maui-The-Magificent · 2026-11-23
- Melt_Dem: Every Layer of Inference Optimization Will Collapse Into a Market — const_reborn · 2026-11-23
- Canary tokens in ~2M AI requests expose breaches at 3 providers and broken ZDR promises — ccerrato147 · 2026-11-23