Merging masked patches speeds up AST training 13.9% with nearly no accuracy loss
kastnerkyle · x · 2026-09-16
Accepted to Interspeech 2026, this paper proposes SpecAugment-Patch Merging, a simple method to accelerate Audio Spectrogram Transformer (AST) training: SpecAugment is first applied to mask input spectrograms at the patch level; after positional embeddings are added, r pairs of masked patches are selected and merged, reducing the number of tokens the Transformer processes.
Experiments show that increasing merged pairs r from 0 to 100 keeps AudioSet mAP nearly unchanged (34.07 → 34.08) while throughput rises from 43.3 to 49.3 samples/sec, a 13.9% relative improvement. The same pattern holds on ESC-50 and Speech Commands V2, with steady throughput gains and only minor accuracy changes—demonstrating faster training at minimal performance cost.
More from Research
- Beyond the Paper: Why Research Artifacts Should Be Science's Basic Unit — valis2400 · 2026-09-21
- BindCraft2 protein design suite launches with full open-source code, free for academic and industry use — mmbronstein · 2026-09-21
- Simpler proof of Komlós conjecture posted to arXiv by Karingula and Lovett — stevenstrogatz · 2026-09-21
- Komlós conjecture solution announced, with overlooked implications for neural network quantization — stevenstrogatz · 2026-09-21
- CIC + LEM surprisingly proves Con(ZF), a breakthrough found with AI tool Fable — leloykun · 2026-09-21
- Qwen presents OmniVChat: synthesized data, benchmark and RL for native audio-visual dialogue — Qwen · 2026-09-21