Listening Forward: Next Patch Embedding Prediction Enables Scalable Audio Learners
Umberto Cappellazzo, Xubo Liu, Stavros Petridis, Maja Pantic
eess.AS, cs.AI, cs.SD
2026-08-20
NAPE pretrains a causal Transformer to predict the next spectrogram patch embedding. NAPE-L hits 50.2 mAP on AudioSet-2M, tying 88M SSLAM at 303M, and reaches 68.8% on IEMOCAP.
Audio self-supervised learning has spent years importing vision recipes: mask-and-reconstruct spectrograms, or distill from a teacher. Pushing the scoreboard then meant piling on machinery. Reconstruction decoders, acoustic tokenizers, EMA teachers, multi-codebook quantizers, extra regularizers. BEATs, EAT, and SSLAM all live on that side.
Language modeling, and more recently vision, trained models to predict the next object instead, a discrete token or a continuous embedding. Audio already unfolds in time, so next-patch prediction is a natural fit. Almost nobody had tried it. NAPE, from Imperial College London and the University of Surrey, asks whether a causal Transformer plus stop-gradient is enough to reach the current first tier.
Waveforms are resampled to 16 kHz mono and turned into 128-bin log-mel spectrograms. A 10-second clip is 128 by 1008, then split into non-overlapping 16 by 16 patches, 504 tokens. A Conv2d stem embeds each patch. The causal Transformer needs a 1D order, and that order decides what counts as the past. Raster walks time first, then frequency. Diagonal mixes the two along anti-diagonals. Time-major exhausts every frequency at one time step before moving forward, and it loses on every benchmark because early tokens see almost no temporal context. Main tables report raster and diagonal.
The encoder is a ViT with causal attention, RoPE applied separately on frequency and time, LayerScale, and query-key normalization. Three sizes: Small 19M, Base 85M, Large 303M. A SimSiam-style three-layer MLP of about 1.8M parameters predicts the next embedding, so the encoder space and the prediction space can differ. The loss is negative cosine similarity, with stop-gradient on the target.
Three pieces have to be on at once:
Drop the shift or the stop-gradient and pre-training diverges. Drop the causal mask and training still runs, but AS-2M falls from 49.6 to 41.8 mAP and ESC-50 from 94.2% to 68.9%. The loss saturates near -1 within about 2,000 steps: the model is copying the answer.
Fine-tuning turns the mask off and uses bidirectional attention over mean-pooled tokens. An appendix ablation says causal plus last-token pooling is only 0.2 mAP worse, so the default is alignment with prior work, not a cliff. AudioSet fine-tuning still uses SpecAugment, Mixup, CutMix, and a weight EMA. The pre-training recipe is thin. The transfer recipe is not.
Pre-training uses unlabeled AudioSet, about 1.96 million unbalanced clips plus 21 thousand balanced. Baseline numbers are taken from the original papers, not a single re-run.
| Method | Params | AS-2M | AS-20K | ESC-50 | IEMOCAP |
| Audio-MAE | 86M | 47.3 | 37.1 | 94.1 | - |
| BEATs iter3 | 90M | 48.0 | 38.3 | 95.6 | 64.5 |
| SSLAM | 88M | 50.2 | 40.9 | 96.2 | - |
| NAPE-B diagonal | 85M | 49.7 | 39.2 | 94.8 | 67.1 |
| NAPE-L raster | 303M | 50.2 | 40.5 | 96.0 | 68.0 |
| NAPE-L diagonal | 303M | 50.0 | 40.4 | 96.2 | 68.8 |
At matched size, NAPE-B still trails SSLAM by 0.5 mAP on AS-2M and 1.7 on AS-20K. NAPE-L ties SSLAM at 50.2 on AS-2M with 303M parameters, and still sits 0.4 behind on AS-20K. Speech emotion is the cleaner win: IEMOCAP reaches 68.8% with NAPE-L diagonal against BEATs at 64.5%, and the Base diagonal run is already 67.1%. Keyword spotting sits near 98% for everyone.
Linear probes are much weaker. The best layer is in the middle of the stack (layer 2 / 6 / 11 for S / B / L). NAPE-L scores 27.1 mAP on AS-2M and 20.4 on AS-20K. The top layers drop another 3 to 5 points, as if they specialized for next-patch prediction. Attention maps attend to the current time column's full spectrum and to earlier patches on the same mel band, a structure the causal objective grew on its own.
This is a clean control for audio SSL. No tokenizer, no EMA teacher, no reconstruction decoder, and fine-tuned classification still lands in the same score band as the heavy recipes. Anyone who wants an audio encoder without mixture supervision and a student-teacher loop now has a simpler starting point.
It does not beat SSLAM at the same size. "Simpler therefore stronger" overclaims it. The accurate sentence is that a causal next-embedding objective can sit at the same table. Frozen features are also a poor product: the probe-versus-fine-tune gap is large.
The paper has no limitations section. Several caveats sit in the protocol.
"Minimal" applies to pre-training only. AudioSet fine-tuning still carries a weight EMA and a full spectrogram augmentation stack. NAPE-L matches SSLAM at roughly 3.4 times the parameter count. Baselines were not reimplemented, so a 0.4 mAP gap cannot be blamed on the objective alone. Pre-training is AudioSet only, with no LibriSpeech-scale speech data, so the IEMOCAP gain cannot be split between the objective and scale. Keyword tasks are saturated. The model learns representations, not a generator of the next waveform.