Inside the zigzag training target: cat direction from 202 sentence pairs, pulled to ±2 SD per layer
Sauers_ · x · 2026-10-20
@Sauers breaks down the gradient design of the zigzag training: it is per-layer, with separate signals for each layer-token pair — maximizing probe readout on even layers and minimizing it on odd ones (or vice versa), targeting a ±2 SD zigzag. The cat direction is computed as mean residual-stream activation over 202 sentences containing "cat" minus the same sentences with 20 other words. Follow-up ideas: a single signal per token, or zig-zagging the entire activation norm instead — easy to learn without disrupting general processing if the cat vector is small in activation space.
Related event: Researcher Trains Qwen3's Internal Representations Into a Zigzag Pattern(9 posts)→
More from Research
- SIGSPATIAL papers benchmark satellite foundation models against costly travel censuses — flosalim · 2026-10-20
- Open-source K-12 math knowledge graph: 674 concepts, 2,843 questions, mistakes traced back to primary school — aigclink · 2026-10-20
- SIGSPATIAL 2026: satellite VLMs recover 98.3% of travel-census accuracy; AlphaEarth wins cross-continental transfer by 33% — flosalim · 2026-10-20
- Gufo, an open-source Strix Halo inference engine, explains its hybrid model caching design — fedeizzo · 2026-10-20
- Metis Reasoning launches with The Age of Reasoning white paper on evidence-grounded AI reasoning — ThoreG · 2026-10-20
- RealityBench Benchmarks VLMs on Documenting Physical Work, Finds Weak Spatial Reasoning — relantic · 2026-10-20