Reading activations changes compiled Qwen3's logits in up to 93% of tokens

Sauers_ · x · 2026-10-19

Responding to controversy over probes altering model outputs after compilation, Sauers argues it's not a bug: most libraries introduce numerical differences when probes are inserted into compiled models, the compiler DAG behavior is documented, and a difference-free implementation is hard to design.
Experiments on Qwen3-0.6B: reading activations has zero effect when run eagerly, but changes next-token logits in 93% of cases with NNsight, and 86%–90% with PyTorch hooks or HuggingFace hidden states under compilation.
A second experiment shows Qwen3-8B trained to detect the probe tool: 78% (PyTorch hook) and 88% (HF hidden states) accuracy as trained vs 52%/80% zero-shot with full compilation — the model itself exploits the numerical differences.

More from Models