Zero-training PCA injection into a 256d null space takes EmbeddingGemma-2 (577M) from #26 to #1 on MAEB audio
JeanMichelRanu · reddit · 2026-10-28
A zero-training experiment on Google's EmbeddingGemma-2 (577M): using only PCA, the author raised its macro score on the 30-task MAEB audio benchmark from 49.3% to 54.4%, moving from Rank #26 to #1 on MAEB audio-only (and #16 to #1 on full MAEB), ahead of 8B-9B multimodal models. Weights: BrunoHays/embeddinggemma-2-nullspace-audio.
Key finding: text-LLM-aligned audio embedders filter their final layers — lexical content survives while paralinguistic traits (timbre, pitch, emotion) are discarded, leaving unsupervised clustering at 1-6%. Probing all 12 Conformer layers showed gender clustering jumping from 0.05% to 66.6% at layers 2-3 and emotion doubling from 19.7% to 39.5% at layer 6.
Method: the terminal projection W (768x512) confines all text vectors to a 512d subspace; SVD reveals 256 orthogonal null directions text never uses. PCA (fitted on 1,400 external audio clips) compresses layers 2 and 6 into those 256 directions: text embeddings stay bit-for-bit identical, retrieval shifts <0.05 points, while gender clustering goes 0.05%→63.1%, artist retrieval 55.2%→93.8%, emotion 19.7%→40.2%.
Trade-offs: no Matryoshka truncation on audio, and normalization slightly shrinks the semantic component, hurting speaker-agnostic tasks (spoken topic 57.2%→21.3%). The author argues this shows how much untapped signal sits in overtrained foundation models, and that actual training on the null subspace could go much further.
More from Models
- Correction: DeepSeek-V4.1-Flash-1017 scores only ~28% on FrontierCode 2.0 — scaling01 · 2026-10-28
- LiveBench update: Claude Fable 5.1 tops real-world leaderboard at 85.7, DeepSeek leads open source — bindureddy · 2026-10-28
- Open-source coding models look benchmaxxed; dev hopes FrontierCode 2.0 stays clean — zyxciss · 2026-10-28
- OpenBMB's MiniCPM-OCR pairs a 1.3B parser with a Jev verifier that scores OCR reliability — omarsar0 · 2026-10-28
- Gemini Reviews Your Chats by Default — the Only Opt-Out Is Deleting All Your Conversations — Kyrannio · 2026-10-28
- Open model demand debate: enterprise self-hosting is overestimated, demand comes from products — zephyr_z9 · 2026-10-28