Turning LLM hard classifiers into tunable soft classifiers with logprobs

JnBrymn · x · 2026-09-24

Inspired by the classification chapter of Hands-On Large Language Models by Jay Alammar and Maarten Grootendorst, the author extends their "hard" LLM sentiment classifier (which just outputs 'positive' or 'negative') into a "soft" one.

Key idea: keep the prompt forcing a single-word answer, then read the logprobs of the candidate tokens from the API response to recover a probability for each class.

Why it matters:

With real probabilities you can set a classification threshold and tune it against a labeled training set, aligning precision/recall with your needs instead of accepting the model's binary verdict.
No fine-tuning required — just a prompt plus a logprob lookup.

The post includes a reusable getsentiment code sample and a 10-minute video walkthrough.

Original post →

Turning LLM hard classifiers into tunable soft classifiers with logprobs

More from coding & agent