ACE extracts invariant algorithmic cores shared across independently trained transformers
burny_tech · x · 2026-11-20
Joshua S. Schiffman's paper introduces Algorithmic Core Extraction (ACE): identifying compact subspaces that are necessary and sufficient for a task and recur across independently trained models. Key findings:
- Markov-chain transformers embed 3D cores in nearly orthogonal subspaces yet recover identical transition spectra.
- Modular-addition transformers form compact cyclic cores at grokking that later inflate under regularization, redundantly distributing computation across equivalent modes — functional redundancy that speeds memorization-to-generalization, yielding an inverse scaling law for grokking time.
- Across six language models spanning two orders of magnitude (GPT-2 Small/Medium/Large, LLaMA-3.1, Gemma-2, Qwen2.5), subject-verb agreement is governed by a single steerable axis aligned across architectures; flipping it inverts grammatical number throughout open-ended generation.
The upshot: training selects behavior, not circuitry, and ACE shifts study from what transformers happen to do to what they must do. Nearly everything runs on a laptop.
More from Models
- Opinion: One Cheap Chinese Open-Weights Model at Parity with Opus 5.5 Means Game Over for Anthropic — IridiumEagle · 2026-11-20
- Gemini Neon reportedly beats Fable 5.2 across the board in informal tests — petrusenko_max · 2026-11-20
- Google cancels Jio-redeemed AI Pro trials citing 'suspicious activity' — craigcpaterson · 2026-11-20
- QuixiInfer: from-scratch Rust engine runs DeepSeek with 256k context on a single RTX 5090 at ~30 tok/s — QuixiAI · 2026-11-20
- GPT 6.2 Sol remains #1 for computer/browser use, says Matthew Berman — MatthewBerman · 2026-11-20
- Microsoft's MAI models have been front-page on Hacker News since Ignite debut — marlene_zw · 2026-11-20