Prototype-Rectified Iterative Self-supervised Manifold Denoising under Severe Acoustic Shift
Ashish Anand Shukla, Rini Smita Thakur, Aryan Das, Vinod K. Kurmi
cs.SD, cs.LG
2026-08-15
Models noise corruption of CLAP embeddings as low-rank affine distortion, compiles closed-form corrections into one static matrix; +12.94 pp on US8K, 0.0009 ms per sample.
Audio-text foundation models like CLAP post strong zero-shot numbers on clean benchmarks, but field deployments (fiber-optic acoustic monitoring, bioacoustic recognition, underwater vessel classification) routinely run below 0 dB SNR, with target events buried under non-stationary interference and reverberation. Zero-shot fails hard: on UrbanSound8K mixed with ten urban backgrounds, LAION-CLAP drops from 78.39% on clean audio to 58.77%.
The three existing adaptation families each fail differently. Gradient-based TTA (entropy minimization in the TENT/SUTA line) feeds confident wrong predictions back into the adaptation loop and effectively trains the model on the noise floor. Prompt tuning needs noise-type annotations that do not exist at inference, plus iterative backpropagation latency. Static geometric alignment is too rigid for environment-specific distortion. In the paper's own measurements, entropy-style adaptation lands within 0.06 pp of zero-shot (58.83% versus 58.77%).
Everything rests on a falsifiable claim, the Affine Noise Hypothesis: after additive waveform noise passes through CLAP's nonlinear encoder, the distortion left in the joint latent space is approximately low-rank affine, a rotation R plus a translation Δ. The evidence is diagnostic SVD on paired clean/noisy embeddings across four environments: over 90% of distortion energy sits in the leading 60 principal components. If the corruption is low-rank, gradients are unnecessary and closed-form linear algebra can invert it.
The clean anchor is free: text prototypes, averaged from 20 prompt templates per class, live in a modality the noise cannot touch.
Calibration (transductive, batch-dependent, 3 rounds) runs three closed-form steps:
After each round, Affine Bias Regression distills the accumulated corrections into a single static (d+1)×d matrix via ridge regression (λ=0.01), always regressing from the original noisy embeddings so error cannot compound across rounds.
Inference is strictly batch-independent: one matrix-vector multiply per sample, 0.0009 ms, no gradients, no batch statistics, no parameter updates.
| Method | US8K Acc | ESC-50 Acc |
| Zero-shot LAION-CLAP | 58.77 | 88.82 |
| TDA | 62.75 | 91.10 |
| PCA++ | 67.88 | 77.57 |
| PRISM | 71.71 | 93.39 |
| ContextDA (oracle noise prompt) | 62.30 | n/a |
US8K means 10 backgrounds × 10-fold CV, 87,320 evaluation instances. PRISM sits 12.94 pp above zero-shot, 3.83 above the strongest training-free baseline (PCA++), and 9.41 above the oracle-assisted ContextDA, which receives the noise type as a privileged prompt. On ESC-50, PCA++ actually falls 11.25 pp below zero-shot; the paper attributes this to its projection erasing signal-bearing directions once SNR is moderate.
The SNR sweep is the most telling result: at -6 dB, zero-shot collapses to 32.69% while PRISM reaches 57.45%, a 24.76 pp gap, with monotonic improvement from -6 dB up to clean audio. Spectrally sparse classes gain the most: airconditioner goes from 22.06% to 64.77%, gunshot from 60.80% to 92.14%.
Component ablation on US8K: OPCA +9.58 pp, CCVD +3.09, ABR +0.27, so cross-modal rotation does the heavy lifting. Cost: 311 ms one-time calibration; 0.0009 ms per sample at inference, 9× faster than PCA++ and orders of magnitude faster than gradient methods (CoNMix around 50 ms). After calibration, accuracy holds at 72.2 ± 0.6% across batch sizes with 80.0% at N=1, provided a warm-up buffer of 128-512 unlabeled samples.
The transferable insight is that the text side of a contrastive model is an underused resource: frozen, free, and immune to target-domain noise, it can serve as a geometric anchor. Nothing in the construction is audio-specific, and the three baselines were themselves ported from CLIP. For edge deployment, closed-form adaptation at one matrix multiply means a new environment no longer requires retraining or prompt tuning.
The low-rank assumption is not universal. On DCASE device-mismatch data (real recording-device shift rather than additive noise), base PRISM falls to 15.63%, below the 17.36% zero-shot baseline: when the hypothesis fails, hard projections destroy signal, and only the CAR variant edges ahead at 17.70%. Deployment needs a prior judgment on whether the distortion is additive noise at all.
The Polyphonic Trap is a failure mode the authors identify and analyze head-on: broadband classes like streetmusic carry semantic variance that overlaps the noise subspace geometrically, so CCVD erases signal and accuracy drops from 92.53% to 75.33%. Confidence-Aware Regression interpolates back to 83.49% at the cost of 0.48 pp overall and reduced gains on sparse classes (airconditioner falls back to 58.35%).
Two reservations from reading it: calibration is transductive, with the matrix drawn from the whole unlabeled test batch or a few hundred warm-up samples, so this is not free online adaptation; and the evidence base is one model (LAION-CLAP), synthetic injected noise from TAU backgrounds, and classification only.