Zero-training PCA head lifts 577M EmbeddingGemma-2 to #1 on MTEB audio, beating 9B models
JeanMichelRanu · reddit · 2026-10-28
A Reddit user boosted Google's 577M-param EmbeddingGemma-2 from 49.3% to 54.4% on the MAEB audio embedding benchmark, jumping rank #26 to #1 and beating 8B-9B multimodal models — with zero training, adding only a 2.37 MB file of frozen PCA matrices.
How it works
- The text backbone is 512d, projected to 768d, leaving 256 perpendicular null-space directions text vectors never touch.
- Early encoder layers (Layer 2, 6) retain speaker, pitch and emotion signals lost in final layers.
- PCA compresses these layers to 256 dims (fitted on 1,400 clips) and injects them into the null space.
Gains: voice/gender clustering 0.05%→63.1%; artist/song audio-to-audio search 55.2%→93.8%; vocal emotion classification 19.7%→40.2%, while keeping 100+ language text-to-audio search.
Caveats: audio embeddings need all 768 dims (no Matryoshka truncation), and unweighted acoustic features hurt some spoken-intent tasks. Weights are open on Hugging Face.
More from Models
- Gemini Reviews Your Chats by Default — the Only Opt-Out Is Deleting All Your Conversations — Kyrannio · 2026-10-28
- Open model demand debate: enterprise self-hosting is overestimated, demand comes from products — zephyr_z9 · 2026-10-28
- Qwenputer: developer compiles a 32-bit RISC-V CPU into Qwen's weights, runs C and MicroPython — teortaxesTex · 2026-10-28
- DeepSeek V5 model IDs spotted in official Harness test suite — zyxciss · 2026-10-28
- Google's AI Mode can't even surface Google's own docs, SEO expert Lily Ray complains — lilyraynyc · 2026-10-28
- Altman's 'alignment was easier than expected' claim clashes with Astra hacking findings — GarrisonLovely · 2026-10-28