Entropy-vector: label-free steering vector that beats temperature, in under 300 lines
voooooogel · x · 2026-09-29
Open-source project entropy-vector builds a steering vector from the gradient of output entropy — no labels or contrast pairs needed.
Key ideas:
- Sample continuations, compute softmax entropy, backprop to the residual stream for an averaged gradient g.
- Pool covariance of hidden states across layers/sequences, take eigenvectors at eigenvalue ranks 16..255, and project g onto that subspace to get v.
- The vector functionally emulates temperature better than temperature itself, with implementations for EOS and repetition control.
Under 300 lines, tested on many models. Example: uv run egv.py --model meta-llama/Llama-3.2-3B-Instruct --site 14 --out v.npz.
More from Research
- Stanford's CME 295 Transformers & LLMs course returns, Lecture 1 on YouTube — afshinea · 2026-09-29
- Stanford CME 295 syllabus: Transformers, MoEs, scaling laws to KV caching — afshinea · 2026-09-29
- An LLM beat Zork by cheating: analysis shows Astra played like it had a walkthrough — rajammanabrolu · 2026-09-29
- USC paper finds 9 LLMs converge on Fourier number features at periods 2, 5, 10 — BrihiJ · 2026-09-29
- C. Elegans Connectome With 302 Neurons Simulated in Real Time on a Smartwatch — bodyaz · 2026-09-29
- New paper links Föllmer process to DDPM denoising diffusion samplers — michaelchchoi · 2026-09-29