Twitter debate: per-token noise training exploits transformer parallel passes, still ~10^4 per update
willcb · x · 2026-10-20
willcb and @9thdev debate a per-token noise perturbation training method on X. @9thdev argues that although per-token noise trades parameter count for layer width and beats weight-space ES by orders of magnitude, the paper's setup still runs 10^4 forward passes per update versus backprop's 3, and the SNR math doesn't change.
willcb counters that it doesn't rely on unique forward passes: it leverages that transformers already compute passes in parallel across many tokens, computing the error term once per token and storing it as a hidden-state perturbation. He adds the approach wouldn't work for FFNs — it merely hides the SNR term in sequence models via a new aggregation axis.
Related event: Debate Over Per-Token Noise Training to Bypass Forward-Pass Costs(2 posts)→
More from Research
- Single neuron steering makes Qwen3-8B insist it's an owl — Sauers_ · 2026-10-20
- Q-CTRL: over 99.5% of qubits sit idle in one fault-tolerant Shor's algorithm blueprint — MJBiercuk · 2026-10-20
- No Siegel zeros proved in just 3 pages, echoing the polynomial method's history — soumitrashukla9 · 2026-10-20
- Logarithmic returns come from opportunities spread across orders of magnitude of cost — DimitrisPapail · 2026-10-20
- Emergent curriculum without labels: prompt weights diverge 1.6x vs 0.75x over training — shangbinfeng · 2026-10-20
- Stackelberg Alignment turns LLM disagreement into a training curriculum via bandit leader — shangbinfeng · 2026-10-20