Symbiotic Architecture for Post-Hoc Audio Extension of Frozen Language Models
Yotaro Kubo, Qi Sun, Yujin Tang
cs.SD, cs.CL, eess.AS
2026-09-25
A CNN injector writes audio into frozen Qwen3-0.6B's KV cache. Test-other WER is 3.91% vs 4.01% for a LoRA ALM; text scores match the original LLM.
A typical audio language model encodes a waveform into vectors and feeds them to the backbone LLM as extra tokens. Prefilling those tokens costs as much as the backbone is wide. Audio sequences are long, so most of that compute is not doing audio work once the LLM gets large.
Fine-tuning the same LLM on audio instructions then hits catastrophic forgetting. The usual fix is to mix old text tasks back into the data, which makes post-training heavier and harder to balance. Sakana AI tries to cut both problems at once: keep audio injection off the full LLM, and never update LLM weights.
The architecture is called symbiotic. Frozen Qwen3 still generates text. A narrower injector writes audio-conditioned vectors straight into every layer's KV cache. At decode time the LLM reads that cache as usual and behaves as an ALM.
The injector has as many layers as the backbone but can be much thinner. Audio prefilling then scales with injector width, not LLM width. A monolithic ALM cannot do this, because audio embeddings have to pass through the full LLM before they become KV.
The injector is a stack of CNNs built from Conformer's LConv blocks: a depthwise convolution between two pointwise layers, with RMSNorm in place of the original normalization. The bet is that audio prefilling cares more about local context, and the backbone Transformer can handle the global part. Swapping in self-attention made training unstable in preliminary runs. A max-pool with stride 8 then maps to keys and values through two affine layers.
Two training tricks decide whether this setup actually converges.
KV Scale Matching forces the injector's K/V magnitudes onto Qwen3's own cache. Key RMSNorm scales are copied from Qwen3; value scales come from the standard deviation of a few calibration sentences. If early attention scores stay too small, gradients never reach the injector.
Noisy RoPE adds a random offset τ drawn from 0 to 256 and a time-scale κ from 0.75 to 1.5 on rotary embeddings, so the injector overfits less to training lengths. Utterances longer than 25 seconds were dropped from training; without this trick, long audio is unusable.
The audio encoder is WavLM-large with LoRA (r=64). Both baselines share Qwen3-0.6B and the same 8× subsample rate. Encoder-only follows SLM: a thin adapter, frozen LLM. Monolithic also LoRA-tunes the LLM (r=16). Training is 90% LibriSpeech (960 hours), with CompA-R, Clotho-AQA, DCASE2025 Task5, and CochlScene filling the rest. AdamW runs for 80,000 steps at batch size 64.
Error rates, lower is better:
| Setup | Params in audio prefilling | LibriSpeech test-other WER | Clotho-AQA error | CochlScene error |
| Encoder-only (frozen LLM) | 1108M | 4.25 | 62.6 | 52.9 |
| Symbiotic (frozen LLM) | 551M | 3.91 | 47.1 | 41.7 |
| Monolithic (LoRA on LLM) | 1108M | 4.01 | 45.4 | 32.7 |
On ASR, symbiotic matches the fine-tuned reference. The four LibriSpeech splits are 1.91/3.73/2.07/3.91 against monolithic 2.16/3.68/2.26/4.01. Audio QA is close (47.1 vs 45.4). Scene classification still lags (41.7 vs 32.7); the authors blame the CNN-heavy injector.
Encoder-only, which only retunes input embeddings, barely steers the frozen LLM on non-ASR tasks. Changing the front door is not enough control.
Forgetting on text is sharper. After audio-instruction fine-tuning:
| Task | Original Qwen3 | After audio FT |
| WikiText-2 BPB (lower better) | 0.88 | 1.32 |
| HellaSwag accuracy | 47.3 | 38.2 |
| GSM8K accuracy | 57.8 | 2.0 |
GSM8K falls from 57.8 to 2.0. Symbiotic never touches weights, so its text scores equal the frozen LLM.
On an H100, batch 4, LibriSpeech dev-other, symbiotic prefills the split in 157.1 seconds against 199.8 for monolithic. The injector subsamples later, so FLOPs are not obviously smaller, but wall-clock still drops. The authors point to fewer kernels and a cache-friendlier parameter size.
Ablations show the tricks are load-bearing. Architecture-only WER on 86 utterances longer than 25 seconds is 120.68%. Noisy RoPE brings that to 10.49%. KV Scale Matching brings it to 4.70%. Without both, long audio does not work.
The usual way to give a text LLM hearing is full or LoRA fine-tuning, then mix data to plug forgetting. This paper treats audio as a write into the KV cache and leaves weights alone. Serving can reuse the existing LLM stack. LLaMA-Adapter also writes a KV prefix, but it still changes the compute graph.
The practical case is a text model you refuse to touch, plus a need for post-hoc audio understanding. At 0.6B, ASR and simple audio QA get close to fine-tuning. Scene classification leans on global acoustic semantics and still favors actually tuning the LLM.
This is an architectural increment, not a new audio encoder and not a bigger training mix. What you save is fine-tuning risk and the parameters activated during audio prefilling, not the total model size.
The authors say this themselves: one small backbone in the compact regime, so the scaling story is unverified. CochlScene still sits about 9 error points behind monolithic.
Total parameters for symbiotic are 1303M, larger than the 1108M baselines. The saving is activated params during audio prefilling (551M vs 1108M), not model size. "No fine-tuning" applies only to LLM weights; the audio encoder still gets LoRA. Training drops clips longer than 25 seconds, and the long-utterance number sits on 86 hold-out files. There is no head-to-head against larger open ALMs, and no audio generation, only understanding tasks.