How the zigzag LoRA was trained: per-layer gradient signals with output forced to match the original model
Sauers_ · x · 2026-10-20
@Sauers details the zigzag training method: a rank-16 LoRA trained across various prompts/questions while forcing answers to match the original model (no token generation change), also minimizing KL from the base model. Training is per-layer, with separate gradient signals for each layer-token pair — maximizing probe readout on even layers and minimizing it on odd layers (or vice versa). The author floats follow-ups: a single signal per token, or zig-zagging the entire activation norm instead, which would be easy to learn without disrupting general processing if the cat vector is small in activation space.
Related event: Researcher Trains Qwen3's Internal Representations Into a Zigzag Pattern(9 posts)→
More from Research
- New paper: let models reward hack in RL, then disable the parameters at deployment — nptacek · 2026-10-20
- Victor Taelin: smart enough compilers will understand and swap your algorithms — bronzeagepapi · 2026-10-20
- Galbot proposes observational scaling law for physical recursive self-improvement in VLM agents — Galbot · 2026-10-20
- ActionRoPE encodes action as world coordinates for spatially consistent game world models — Zhiyang Deng · 2026-10-20
- UniAVC unifies audio-video generation via progressive training and asymmetric noise scheduling — Kien T. Pham · 2026-10-20
- AHE paper auto-evolves coding-agent harnesses with decoupled files, lifting Terminal-Bench 2 to 77% — morgymcg · 2026-10-20