Geiping team trains LLMs to report internal MoE routing

Jonas Geiping's team has released a new preprint they describe as "a bit weird," proposing to train LLMs for "digital proprioception": having the model verbalize its own internal state—specifically, reporting which MoE expert was activated for a previous token at a given layer. Their current finding is that the method trains well and holds up under intervention experiments; what makes it notable is that it provides a verifiable subset of the otherwise hard-to-validate introspection research that neutral observers can check.

Confirmed

• Core method: routing decisions in MoE models don't exist in any offline training data or RL signal, so models normally cannot know them; the researchers picked up real-time labels directly from the model's forward pass and trained the model to verbalize these internal states, with good results.
• Motivation: introspection is hard to study in both humans and LLMs, and most introspection questions rely on unverifiable self-report; this kind of "proprioception" is one of the few internal signals that a neutral observer can verify.
• Robustness experiment: after ablating the top-1 (strongest-activated) expert in the MoE model—which genuinely changes the forward computation—the task becomes harder, but the trained model still correctly reports the new top expert; since the input text is completely unchanged and the change happens only inside the model, this shows the model truly senses and verbalizes changes in its internal weights.

Why it matters

• This work moves the question of "can a model know what it's doing" from philosophical debate based on self-report into experimentally testable territory: routing decisions provide objective ground-truth labels.
• The ablation experiments show the model's reports are not memorization or inference from input, but a genuine readout of internal computational state, opening a viable new path for introspection and interpretability research.

2026-10-21 ~ 2026-10-21 · 6 related posts

• [source] Teaching LLMs digital proprioception: models learn to report their own internals — jonasgeiping · 2026-10-21
• Researcher Proposes Giving Language Models 'Proprioception' to Make Introspection Verifiable — jonasgeiping · 2026-10-21
• Why train models to report their internals? A verifiable subset of introspection — jonasgeiping · 2026-10-21
• [source] Live labels from the forward pass teach MoE models to report their own routing — jonasgeiping · 2026-10-21
• [source] Even after ablating the top MoE expert, models can still verbalize the new one — jonasgeiping · 2026-10-21
• Preprint: models can still verbalize internal changes after ablating their top expert — jonasgeiping · 2026-10-21