150M-parameter guardrail model beats DeepSeek V4 Flash and GPT-6-Luna at off-topic detection
kalyan_kpl · x · 2026-10-18
A developer released Ammu 1.0, a family of off-on-topic guardrail models that classify whether a user query matches the system prompt before it reaches the LLM, cutting API costs and latency.
Why not use LLMs for off-topic blocking
- Classifying every query with an LLM is expensive
- Large models add latency
- LLMs struggle with medium and hard off-topic cases
Results (from the chart)
- Ammu 1.0 base (150M params) hits 91.56% overall Macro-F1, beating DeepSeek V4 Flash (81.95%) and GPT-6-Luna (79.12%); small/tiny/nano variants (68M/32M/17M) reach 90.35/90.87/87.31
- Trained on a high-quality synthetic dataset spanning easy, medium, and hard instances
- Small models clearly outperform both LLM APIs on medium and hard queries
More from Models
- Paying Google AI subscriber blocked: custom-domain Workspace accounts can't access AI Studio — ChrisRackauckas · 2026-10-18
- Running Ling 3.1 Flash (560B MoE, 25B Active) Across 4 DGX Sparks via llama.cpp RPC — BerryBlush_Rogers691 · 2026-10-18
- Users press OpenAI on long-delayed Adult Mode a year after it was promised — ___Patrice___ · 2026-10-18
- Polymarket opens betting market on who releases the next Gemini Argon model — Polymarket · 2026-10-18
- Grok matter-of-factly lists which groups user was racist against, exempting Africans and Latin Americans — teortaxesTex · 2026-10-18
- A quick test for benchmaxxed demos: swap in Mirror's Edge and see if it generalizes — teortaxesTex · 2026-10-18