Voice Memory for Agentic Speech Recognition
Chao-Han Huck Yang, Zih-Ching Chen, Piotr Zelasko, Zhehuai Chen, Jagadeesh Balam, Boris Ginsburg
cs.CL, cs.AI, cs.SD, eess.AS
2026-07-29
Voice Memory adds a human-readable per-domain memory.md to ASR correction: a frozen corrector decides per utterance whether to act or abstain, and edits land only when they strictly improve a held-out score. Weighted WER drops 8.36% to 7.52%, harmful edits 64% to 35%.
Speech recognition has entered a low-error regime, below 2% WER on clean speech, but decoders still make systematic, domain-specific errors. The conventional fix is generative error correction (GER): hand the n-best hypotheses to an LLM and let it rewrite.
The trouble shows up at low error rates. The real question is no longer whether the model can fix an error but whether it should touch the hypothesis at all. Strong LLMs over-correct, editing tokens that were already correct, turning "home is" into "home's", normalizing digits and spelling to their own conventions. Static GER ends up raising WER above the unedited 1-best on low-headroom domains.
Voice Memory is an inference-only scheme whose core idea is to store the correction skill in a human-readable text file, memory.md, instead of in the weights. It calls this the listener-thinker architecture, extending the classic ASR-LM cascade.
The listener runs synchronously: a frozen decoder turns audio into evidence, and a frozen corrector reads memory.md and decides per utterance whether to act (apply a correction) or abstain (keep the 1-best). The thinker runs asynchronously: a separate optimizer revises memory.md from scored rollouts, making only bounded edits and accepting one only when it strictly improves a held-out score.
The two roles couple only through the memory, so no weights change and the skill stays auditable and portable across corrector families. The optimizer is forward-pass only, no gradients. Compared with fine-tuning or LoRA, it needs about two orders of magnitude less data (roughly 10^2 utterances versus 10^3 to 10^4), produces an artifact under 10 KB, and is reversible.
Main results on ten HyPoradise domains with an open corrector (Qwen3-30B-A3B): weighted WER drops from 8.36% to 7.52% (7.47% with three in-context examples), with no dataset regressing below its 1-best baseline. The harmful edit rate on financial news falls from 64% to 35%. Gains concentrate where headroom is largest: air-travel commands 8.40% to 3.40%, far-field CHiME-4 12.69% to 10.46%, and the recoverable-information ratio correlates with 1-best WER at r=+0.90.
The memory transfers across corrector families: swapping in a different frozen corrector (Claude-4.6-Sonnet), the same memory still works, and on ATIS the self-formed memory beats even the n-best oracle (rho=1.28).
The headline claim is that a better agent here is simply a better text file. Domain adaptation stops being a weight-change problem and becomes editing a human-readable rules document, which makes it auditable, portable, and reversible, with zero new parameters and no added latency on the inference path. For ASR deployment teams, customizing a domain no longer means retraining or bolting on a LoRA: maintain one memory.md per business domain, and a new domain goes live in minutes rather than hours.
The main analysis uses a single corrector (MiniMax-M3; the appendix reproduces with Qwen3-30B-A3B), and the rho and benign-mass metrics depend on the sentence encoder used (all-MiniLM-L6-v2, 4-bit). Residual WER is not the same as residual headroom: on LibriSpeech-Clean, 42.9% of residual errors are forgivable out-of-vocabulary names and spelling variants. The self-formed memory loses on number-formatting-heavy WSJ (data-hungry) and has nothing to learn at the LibriSpeech floor. There is also a tension in the framing: "no domain regresses" is the selling point, but on the lowest-headroom domains rho sits near zero or negative, meaning the mechanism is largely holding still there rather than genuinely correcting.