Do Models Already Know They'll Comply?
its_vayishu · x · 2026-07-17
This deep-dive thread analyzes a paper exploring whether models internally know if they will follow an instruction before generating output.
The core argument is that before a model outputs a token, its hidden states already linearly encode whether the current rollout will "comply". This suggests the decision signal forms very early in the forward pass, rather than emerging at the final logits.
However, the thread also raises a critical caveat: the paper's sensitivity analysis found that "phrasing changes" mattered more than "task familiarity" or "instruction difficulty." Yet, because these phrasing tweaks only altered surface forms while preserving semantics, it leaves two indistinguishable explanations:
- The change simply made the constraints easier for the model to parse;
- The change genuinely altered the model's internal representation of its "commitment to comply."
Ultimately, while the findings are thought-provoking, they aren't quite conclusive.
More from Research
- AI folks, stop saying "biology" when you mean drug development — KevinKaichuang · 2026-10-26
- ostris pre-training "Primal VAE": 1-channel latents match modern 8x VAEs via patching — ostrisai · 2026-10-26
- Shanghai AI Lab-led team releases 100-hour egocentric wearable dataset for humanoid training — ChuckDBrooks · 2026-10-26
- Flow Reasoning Models turn flow matching into iterative reasoners, hitting 99.5% on Sudoku-Extreme — alec_helbling · 2026-10-26
- Surprising find: Alzheimer's-linked amyloid fibrils deliver RNA to sperm for embryo translation — EricTopol · 2026-10-26
- HarnessMM cuts agent tokens per task by up to 88% while raising success rates — omarsar0 · 2026-10-26