From Masking to Merging: Rethinking SpecAugment for Efficient Audio Spectrogram Transformer
Minhee Park, Hyowon Ahn, Chanwoo Kim
cs.SD, cs.LG, eess.AS
2026-09-06
Merges SpecAugment-zeroed AST patches by dimension-wise max. At r=100 on AudioSet, mAP stays 34.07 to 34.08 while throughput rises 13.9%, from 43.3 to 49.3 samples/sec.
The Audio Spectrogram Transformer (AST) treats a Mel spectrogram as an image: 16x16 patches become tokens, and self-attention scales with the square of the token count. SpecAugment is standard during training. It punches random time and frequency stripes out of the spectrogram so the model has to rely on context. Those zeroed regions carry almost no spectral content, yet their embeddings still run through every Transformer block.
Existing token-reduction recipes either drop patches at random (PaSST's Patchout) or merge similar tokens (ToMe, FastAST). Both need extra machinery. This paper asks a narrower question: SpecAugment already marks the uninformative locations. Use those marks as merge candidates.
Both steps happen before the Transformer encoder.
Positional embeddings go on first so the surviving tokens still carry time and frequency location. At r=0, mAP is 34.07±0.18 against 34.11±0.35 for vanilla SpecAugment, so grid-aligned masking does not itself hurt the augmentation. Among max, mean, sum, and random drop, max won on stability (34.10±0.31 mAP at r=90) and is used everywhere else. Throughput is nearly identical: 48.5 samples/sec for drop, 48.63 for max.
Candidates come only from fully masked patches, so the cap on r tracks how many zeros SpecAugment produces: roughly 212 on AudioSet, 110 on ESC-50, 34 on Speech Commands V2. Frequency masks span at least two rows to keep a usable candidate pool. Vanilla SpecAugment cuts continuous frames, so a patch can land half-masked after the split; grid alignment removes those mixed patches and leaves a clean merge set.
The backbone is ImageNet-pretrained DeiT-Base distilled (87M parameters), one RTX 4090. AudioSet experiments use the 22k-clip balanced subset, not the 2M-clip full set, for 25 epochs at 5e-5 with weight averaging from epoch 6 to 25. Mixup is on for AudioSet and Speech Commands V2 at ratios 0.5 and 0.6. Throughput is wall-clock over the full training step.
| Setup | mAP / Acc | Throughput (samples/sec) | Memory |
| AudioSet r=0 | 34.07±0.18 | 43.3 | 24.64 GB |
| AudioSet r=100 (16.5%) | 34.08±0.24 | 49.3 (+13.9%) | 21.50 GB (down 3.14) |
| PaSST-U at 16.5% | 29.58±0.27 | 46.7 | 21.00 GB |
| ESC-50 r=0 to 50 | 89.20 to 88.67 | 127.3 to 142.9 | not reported |
| Speech Commands V2 r=0 to 15 | 98.13 to 98.06 | 411.1 to 429.4 | not reported |
PaSST-U was retrained under the same optimizer, 16 kHz sampling, and balanced AudioSet. The original PaSST paper used 32 kHz audio and full AudioSet, so absolute mAP is not comparable. The efficiency comparison is: at a matched 16.5% reduction, this method hits 49.3 samples/sec versus 46.7 for PaSST-U. FastAST has no public code, so there is no controlled comparison. ESC-50 accuracy drops 0.53 points in exchange for 127.3 to 142.9 samples/sec; keyword spotting stays almost flat.
This is an incremental training trick, not a new architecture. For pipelines that already train AST with SpecAugment, the change is small: align masks to patches, merge after positional embeddings. No similarity module, no change to the inference graph.
The speedup is training-only, and only while SpecAugment is on. Inference usually turns augmentation off, so this does not help serving. Porting the idea to other audio ViTs assumes those models already mask time and frequency stripes at train time.
The authors note the obvious cap: candidates are fully masked patches only, so AudioSet tops out around 16.5% reduction. They do not report full AudioSet or inference latency. PaSST-U's mAP sits about 4.5 points below AST, some of that from implementation and data choices, so the table does not show that merging is more accurate than Patchout. Throughput is wall-clock on one 4090. Merging happens once at the encoder entrance, unlike ToMe's layer-wise schedule.