Researcher Trains Qwen3's Internal Representations Into a Zigzag Pattern

@Sauers published and open-sourced an interpretability experiment: using rank-16 LoRA on Qwen3-1.7B, the model was trained to raise the internal "cat" representation on even layers and suppress it on odd layers (or vice versa) according to prompts, making activations follow a zigzag path across layers—while the model's answers to normal questions remained identical to the original. The experiment code, charts and full findings are available in the repo SauersML/zigzag.

Confirmed

• Training method: rank-16 LoRA trained across varied prompts/questions, forcing answers to match the original model (leaving token generation unchanged) while minimizing KL divergence from it; training proceeds per layer, with independent gradient signals for each layer-token pair—maximizing probe readings on even layers and minimizing on odd layers (or the reverse), with the target curve pulling the "cat direction" layer by layer into a zigzag at ±2 SD. The base model cannot do this; after fine-tuning, the model can precisely trace the instructed zigzag in activation space.
• The "cat direction" is defined by the difference between 202 sentences containing "cat" and a baseline.
• Ablations located the trigger mechanism: removing "cat" from the prompt still yields the zigzag; what's truly required are the "even"/"odd" tokens—the word "even" alone can induce the zigzag; and only ablating relevant components in the first few layers (0-3) has any effect. The author notes this behavior does not generalize to other cat probes.
• Unexpected side effect: after having the model raise the "cat" representation on even layers to answer questions, then asking "name an animal, one word," 46% of answers were cat (95% CI 40–53%); when raising cat on odd layers, only 10% (CI 5–15%). This behavior was never taught during training.
• The experiment is open-sourced (repo SauersML/zigzag), including code, charts and key findings in the README; the author says results aren't perfect and plans to add training data and train new models for generalization, teasing cooler generalization experiments to come.

Why it matters

The experiment shows that a large model's internal representations can be steered directly via natural-language prompts (a kind of "internal representation instruction"), and that this steering influences subsequent behavior (a large shift in the probability of voluntarily answering "cat") without changing surface outputs—offering a reproducible toy-scale example for controllable interpretability research.

2026-10-20 ~ 2026-10-20 · 9 related posts

• You can train a model to zig-zag its activations, mini interpretability demo — Sauers_ · 2026-10-20
• Inside the zigzag training target: cat direction from 202 sentence pairs, pulled to ±2 SD per layer — Sauers_ · 2026-10-20
• How the zigzag LoRA was trained: per-layer gradient signals with output forced to match the original model — Sauers_ · 2026-10-20
• [source] Untrained side effect: after the cat-zigzag request, the model names 'a cat' 46% of the time — Sauers_ · 2026-10-20
• Qwen3-1.7B LoRA-trained to zigzag its 'cat' representation across layers — but it just tracks the word 'even' — Sauers_ · 2026-10-20
• Ablating the zigzag: knocking out 'cat' changes nothing, only the tokens 'even'/'odd' matter — Sauers_ · 2026-10-20
• Ablation: even/odd tokens, not 'cat', control the zigzag — and it fits only one probe — Sauers_ · 2026-10-20
• [source] Researcher trains Qwen3-1.7B to obey activation-space instructions, drawing zigzag cat representations — Sauers_ · 2026-10-20
• [source] SauersML open-sources the zigzag repo: Qwen3-1.7B trained to zigzag its cat representation on request — Sauers_ · 2026-10-20