A 414k-param transformer learns boid flocking; probes read the rules — with a shortcut trap
rage_81 · reddit · 2026-10-11
The author built a from-scratch transformer (one token per bird, two attention layers, 414k weights, laptop CPU) to predict boid flock movements, then tested whether probe-readable rules are the ones actually used. Findings: R² 0.990–0.994 on held-out clips across 4 runs, generalizing to 50 birds from 12 trained; a baseline without attention/positions gets only R² 0.40. Linear probes easily read all three boid rules — cohesion even before training (R²≈0.85) — and alignment vanishes in all 4 runs when the second layer is forced to self-attend. Surprise: the first model wasn't using the flock at all, since 8 ticks of history leaked the prior answer into the input. Open question: whether forced self-attention is a fair test. Site and code are open.
More from Research
- OpenTac auto-routes tactile sensor PCBs from a 3D model in 3 seconds, covering 93% of surface — anand_bhattad · 2026-10-23
- arXiv paper gives StopGrads theoretical grounding: convergence without an underlying loss — srush_nlp · 2026-10-23
- PEARL demo: an LLM agent that checks experiment results, revises plans, and re-tests — KevinKaichuang · 2026-10-23
- Berkeley's StyleStream 2.0 cuts streaming voice conversion latency to 520ms, 2x faster — Berkeley · 2026-10-23
- Math PhD student quits academia: AI is making math unrecognizable — pickover · 2026-10-23
- ICLR Submission Found Containing LLM's Own Meta-Comment Admitting Results Weren't Significant — _onionesque · 2026-10-23