VoxMem: Benchmarking Multimodal Memory in Large Audio Language Models
Yang Xiao, Vidhyasaharan Sethu, Eun-Jung Holden, Ting Dang
eess.AS, cs.SD
2026-09-26
VoxMem scores 15 LALMs on 3,196 multi-session spoken-memory items. At 32K none exceed 40% (best 38.5%); models keep what was said far better than who, how, or what was audible.
Spoken assistants have to remember across visits, and current tests mostly check the words. SpokenWOZ and ContextDialog score lexical content. AudioMarathon, LongSpeech, and VoiceGiraffe stretch a single recording. Audio MultiChallenge stays inside one dialogue, three to eight turns. Who spoke, how they sounded, and what was audible in the room live only in the waveform. They vanish in a transcript. Memory operations are equally thin: most items are single-shot retrieval. Combining evidence across sessions, tracking a changing state, and refusing when the history cannot support an answer are barely covered.
Real use is also multi-session. Topics shift, speakers recur, days pass. A long single file tests whether the model understood that file, not whether it can bind evidence across separate episodes. Length studies often swap the question when they change the duration, so length and difficulty move together. VoxMem scores two axes together: which acoustic evidence must be recovered, and which operation must be applied to it.
Researchers at the University of Melbourne and UNSW lay the task on a 4×4 grid and drop the Speech-Semantics × Information-Extraction cell. Both sides are lexical, and the item would collapse into dictation. Fifteen valid cells remain.
Speech semantics covers facts, plans, and preferences that a transcript already contains; it is the lexical baseline. Speaker identity puts one assistant in front of several equal-status users, so the answer is bound to a voice and names are banned from the text. Paralinguistic cues cover hesitation, surprise, rate, and laughter. Environmental sound covers traffic, alarms, appliances, weather. The last three types are audio-native by construction.
Information extraction (IE) pulls from one past session, either locating it by content then reading an acoustic attribute, or locating it by an acoustic cue then reading a fact. Multi-session reasoning (MSR) counts, matches, or aggregates; permuting evidence sessions must not change the answer. Temporal evolution tracking (TET) asks for the latest state, a state at a named time, or the whole update chain. Order matters, and acoustic changes are never announced in words. Answer refusal (AR) is derived from an answerable item by removing or neutralizing the minimum evidence for a unique answer. Of 166 AR variants, 130 survived full validation.
Each history mixes evidence sessions, haystack sessions that share the topic or cue but the wrong value or speaker, and InstructS2S filler. The question, gold answer, and evidence stay fixed at 8K, 16K, 32K, and 64K; longer budgets only add haystacks and fillers. Length is measured with a Whisper encoder at 50 tokens per second, about 2.5 to 20 minutes of audio, so the axis does not depend on any model's tokenizer. Evidence and haystacks are spread through the timeline.
User turns are fully synthetic: Higgs-TTS-3, 30 fixed VCTK voices, style tags for paralinguistics, ESC-50 clips mixed at 10 dB SNR. Assistant turns are text, so the assistant cannot leak acoustic answers and models that do not generate speech can still take the same history. Quality control uses Gemini-3.7-Flash, which is not in the evaluated set. The release has 799 questions, 3,196 items, 34,743 sessions, 177 hours, across 20 everyday topics.
A transcript-only check on 669 answerable questions at 8K sends Gemini-3.7-Flash from 46.2% to 4.4% on audio-native items, and only from 75.9% to 71.0% on speech semantics. A text classifier that tries to separate evidence from haystack sessions lands at 51-56%, against 47-49% with shuffled labels.
Fifteen LALMs take user audio through native interfaces, with no ASR front end. Gemini-3.7-Flash scores free-text answers without seeing the history. Re-judging with GPT-5.6-Luna gives κ=0.953 and 98.0% agreement. Capability refusals that claim no audio access count as wrong.
At the 32K reference budget, no model exceeds 40% overall. Qwen3.8-Omni-Flash leads at 38.5%, then Gemini-3.1-Pro at 35.9% and Gemini-3.8-Flash at 34.0%. Five proprietary systems average 33.0%; ten open-weight systems average 21.9%.
| Setting | Proprietary mean | Open-weight mean |
| Overall 8K | 40.1% | 26.6% |
| Overall 32K | 33.0% | 21.9% |
| 32K speech semantics | 55.6% | 31.6% |
| 32K speaker identity | 32.7% | 26.7% |
| 32K paralinguistic | 20.0% | 14.5% |
| 32K environmental | 21.9% | 15.5% |
Difficulty lives in the operation-evidence pairing. Mean accuracy at 32K across 15 models: speech-semantics TET 44.5%, speaker TET 20.0%, paralinguistic TET 3.4% (n=49), environmental TET 1.2% (n=32). IE on paralinguistic cues is 8.8%. MSR is not the harder cousin: speaker MSR is 43.1% against 29.0% for speaker IE. Refusal flips the ranking. Paralinguistic and environmental AR sit at 34.3% and 34.2%; speech semantics and speaker identity sit at 19.9% and 16.2%. High refusal on acoustic items mostly tracks inability to use those cues.
The same questions and the same evidence, with more surrounding sessions, lose points. Proprietary overall accuracy falls from 40.1% at 8K to 33.0% at 32K; open-weight from 26.6% to 21.9%. By 64K, models keep 70.3% of their 8K score on speech semantics, 69.8% on paralinguistic cues, 66.5% on speaker identity, and 63.2% on environmental sound. Paralinguistic memory is weak at every length and decays about as fast as lexical memory. Speaker and environmental memory erode more as the history grows.
Error attribution at 64K, attempted answers only. Binding errors are 46% of IE mistakes: the cue was recovered, the session was not. MSR splits 48% localization and 32% operation execution. TET is 55% localization and 5% execution; tracking dies before the temporal reasoning starts. Speaker errors are 48% binding. Paralinguistic errors are 63% localization. Environmental errors split 41% localization and 39% unsupported answers. The three audio-native types do not share one bottleneck.
For anyone building a voice agent or a long-context audio model, this is a yardstick that separates word memory from sound memory. Stretching the window does not freeze access: the item stays put, the haystack grows, accuracy falls. A pipeline that transcribes then feeds an LLM will score near zero on the audio-native cells, because the answer is not in the words.
This is a benchmark, not a new memory architecture. Code and data are public. If the next system claims cross-session voice binding, vocal-state tracking, or ambient-sound memory, this is the pressure test that actually asks for those things. Synthetic speech, text assistant turns, and long context only: that is how far the numbers can travel, and they do not stand in for a live call.
There is no standalone limitations section. All user speech is TTS. Paralinguistics are style tags. Environment is a 10 dB mix. That is not a real call, overlapping talk, or a live room. Assistant turns are text, so the test is hearing the user and reading past replies, not full speech-to-speech memory.
Questions were written by Gemini-3.7-Flash and GPT-5.6-Luna; the same Gemini family also runs QC and the primary judge. A second judge reaches κ=0.953, but the biggest AR disagreements are capability refusals: of 50 items the second judge credits and the production judge rejects, 32 claim no access to prior conversation. The length axis is Whisper's 50 tokens per second, not any model's real spend, and 64K results use a separate model subset. There is no retrieval or external-memory comparison, so the claim holds only for stuffing the whole history into context. English, 20 everyday topics, 30 VCTK voices. The haystack probe is a TF-IDF logistic regressor.