willcb: per-token noise training won't work for FFNs, relies on sequence aggregation axis
willcb · x · 2026-10-20
In a follow-up, willcb clarifies the scope of the per-token noise perturbation training method: it wouldn't work for FFNs, and only manages to hide the SNR term in the context of sequence models by using a new axis for aggregation. This bounds his earlier claim that the method exploits transformers' already-parallel forward passes, computing the error term once per token as a hidden-state perturbation.
Related event: Debate Over Per-Token Noise Training to Bypass Forward-Pass Costs(2 posts)→
More from Research
- Two New Robotics Benchmarks Released: MicroPlanBench and MicroAnnotationBench First Rankings — chris_j_paxton · 2026-10-20
- ICLR reviewer despairs: 5 assigned papers nearly all AI-written, calls peer review broken — peterbhase · 2026-10-20
- Meta: Co-Design for AMD GPUs Delivers Up to 1.5x Training Throughput vs Direct NVIDIA Ports — burny_tech · 2026-10-20
- Continuous Softmax Attention work shared, drawing attention — burny_tech · 2026-10-20
- Muon optimizer tweak with a polynomial grid saves ~0.5% of LLM training compute — burny_tech · 2026-10-20
- HeadEdit steers LLM behavior via frozen unembedding matrix, no fine-tuning needed — burny_tech · 2026-10-20