Supercharging LLM classification with logprobs: soft classifiers you can threshold-tune

JnBrymn · x · 2026-10-28

An Arcturus Labs blog post extends the classification chapter of Hands-On Large Language Models: instead of a "hard" classifier that only outputs positive/negative, read the logprobs to build a "soft" classifier returning per-class probabilities.
Soft outputs let you set a threshold to optimally align predictions with a training set, tuning the precision/recall trade-off.
The post walks through the hard-classifier baseline (task prompt, text to classify, strict output-format constraint), then shows the logprob-reading trick on the first token's distribution.
In the thread, JnBrymn notes that "System 1" decision models are essentially LLMs with reasoning off and a single token predicted—so frontier models' single-token probability behavior deserves retesting; cmcollier says the work was inspired by a surprise zero-shot LLM result in a special domain.

More from coding & agent