No retraining needed: an LLM agent fixes a TTS model by editing weights directly
N1kuson · reddit · 2026-11-06
Building Silencio-2 (a Kyutai Pocket TTS finetune), the author's <whisper> tag only worked 1 in 5 takes. Instead of collecting more data, an LLM agent (Claude Opus 5.5) probed the weights: tag embeddings barely drifted from their seed words (60–80% cosine), audio frames attended to <happy> 100–3000× more than average tokens but other tags only 1–10× — the model understood the tags but barely read them.
The fix: per layer, a rank-1 term added to WK — a least-squares tag detector plus a key shift along the slowest RoPE pairs. 524k weights touched (3.4%) but only 5k free parameters, zero gradient steps. Whisper success jumped (attention boost alone gave 62% at ×30), 7-class emotion separability 0.45, untagged WER 0.006.
Caveats: some takes glitch mid-sentence, whispers can slur (WER 0.18), <surprised> pitched up 4 semitones. Boosts ≥×14 smear words. Pushing embeddings away from seed words didn't work; audio-frame-only boosts mostly vanished.
The companion image shows the same weight-space editing approach reading a Breakout policy network (99% confidence, no tools), one-shot fixing CartPole (0→100%), and lifting Qwen3.5-0.8B from 7.9% to 48.1% on MATH-500 with no training.
More from coding & agent
- GradiumAI open-sources Phonon: fully on-device TTS for voice-enabled agents — mattturck · 2026-11-06
- Fluency, not latency: HRI metrics offer a better benchmark for real-time AI agents — Stefania_druga · 2026-11-06
- LangChain: graph-based agent memory lifts connected-question accuracy 27%, cuts cost 58% — JeremyCMorgan · 2026-11-06
- Void Unifies Build and Deploy in One Command, With Admin-Controlled Coding Agent Access — cnakazawa · 2026-11-06
- After 610 Agent Commits in 4 Weeks, This Dev Replaced Human Code Review With Agent Reviewers — ShoddySherbert8502 · 2026-11-06
- Cursor launches live in-editor app building with one-link sharing, from idea to deployed app — mattyp · 2026-11-06