Using raw digit logits from llama.cpp as a fast classifier without grammar constraints
rhinodevil · reddit · 2026-09-19
A developer shares a lightweight classification trick implemented in mtllm, built on llama.cpp: instead of using the grammar feature, prompt a small LLM (9B or smaller) with a numbered multiple-choice question and stop inference right after the first real token is generated.
- Read raw logits of digit tokens only, merging probabilities of variants like " 2" and "2"
- Normalize so the candidate digits sum to 1.0, then pick the highest as the classification result
- Faster and simpler than grammar constraints; works for yes/no questions too, and can be extended with a "None of these" option to exceed 10 classes
The author notes this also lets you force an LLM to answer any multiple-choice question. Full implementation details are in the linked source code.
More from coding & agent
- EvoSkill v2 treats agent skills as executable state—and found agents learning to cheat — rohanpaul_ai · 2026-09-19
- Musecases Launches: A Community-Voted Prompt Library for AI Agents — ChrisUniverse · 2026-09-19
- ~40,000 passing tests: dev explains why he doesn't review every line of AI code — doodlestein · 2026-09-19
- NVIDIA's SoL-Pi GitHub repo: auto-research loops for efficient agent harnesses — aigclink · 2026-09-19
- Blogger: most engineers are just vibe coders, real skills lie in fundamentals — ashishllm · 2026-09-19
- Installers are going away — prompts with installer skills are next — BLUECOW009 · 2026-09-19