Entropy trajectory shape predicts Qwen3-4B errors and transfers to unseen tasks

Happy_Brilliant7827 · reddit · 2026-09-06

Core finding

The author ran calibration experiments on a local Qwen3-4B-Instruct-2507 (Q5KL): 60 coding tasks, 108 unique generations (20 failures), capturing 170 entropy/logit-derived statistics per generation in a single pass, with Benjamini-Hochberg correction.

Surviving signal: a trajectory-shape statistic (d = −0.94, adjusted p = 0.008) — when the model's entropy plateaus early and stays flat, the generation is likely wrong.
Standard metrics fail: the commonly reported maxentropy gives only d = +0.17, p = 0.39. Aggregate entropy destroys the information that predicts correctness, as the Entropy-Lens paper predicted.
Transfer: across 300 task-grouped random splits holding out entire tasks, 0.73 mean balanced accuracy on unseen tasks, above chance in 92% of draws — the signal learns what wrongness looks like for this model, the exact question raised by DeepMind's semantic entropy paper.

Honest negatives

The same method found nothing usable on Granite-4.1-3b, and its task-level holdout caught a false positive (0.87 on training, 0.47 below chance on unseen tasks) — the method catches its own failures.
On a different task bank, the effective signal shape differed (an early spike predicting correctness): the signature is per-model and per-task-set. No universal entropy rule; custom tuning is required but the signal is still valuable.

Full data and live reports (both holdout levels, power analysis, intervention ranking) are publicly available.

Original post →

Entropy trajectory shape predicts Qwen3-4B errors and transfers to unseen tasks

More from Models