Reinforcement Learning with Evolving Rubrics as Rewards for Audio Reasoning
Fangxu Yu, Tao Feng, Dehai Min, Zinan Lin, Weijia Xu, Michael Xu, Philip S. Yu, Ge Liu, Tianyi Zhou
cs.SD, cs.CL
2026-08-04
AudioRubrics turns per-sample rubrics into rewards that evolve with the policy, lifting Qwen2.5-Omni-7B to the top of three audio-reasoning benchmarks by scoring reasoning against audio-grounded criteria.
Training large audio-language models (LALMs) with reinforcement learning usually means rewarding only the final answer (an outcome reward). That leaves a hole: a model can parrot the right multiple-choice letter from language priors without genuinely listening to the audio. Process rewards, which score the reasoning itself, fix half the problem but rely on a fixed, hand-written rubric that ignores which question is asked, never checks the reasoning against the acoustic evidence, and saturates the moment the policy learns to satisfy it. Audio questions are not uniform: some hinge on perception, others on multi-step inference, and one static yardstick fits none of them for long.
AudioRubrics adds an evolving rubric reward on top of the outcome reward. Three ideas do the work.
First, the rubrics are anchored to sound. Instead of handing the judge a transcript, the system feeds the raw waveform to a single audio-capable model (Gemini-3.1-Pro) that doubles as rubric generator and judge. For each question it writes a handful of binary criteria, each required to cite concrete content actually present in the clip, with weights summing to one.
Second, the rubrics evolve with the policy. At each step the model samples eight rollouts; the judge contrasts them, drops criteria that every rollout passes or fails (variance filtering, since they carry no group-relative signal), and distills harder criteria from where strong and weak rollouts diverge, including negative criteria that capture recurring failure modes. Survivors carry into the next iteration, so the bar only rises and keeps targeting the policy's current weakness. The share of freshly evolved rubrics climbs from about 35% early in training to roughly 60% by the end.
Third, a length penalty stops the model from padding its reasoning to satisfy more criteria. Process rewards naturally favor longer traces, which decay into redundant, circular, or hallucinated chains. A linear penalty on reasoning length counterbalances the rubric term: one rewards substance, the other bounds volume.
The per-rollout reward is a weighted sum of outcome reward, rubric reward, and length penalty, optimized with GRPO.
The base model is Qwen2.5-Omni-7B, trained with GRPO for 400 steps on four H100 GPUs on about 40k AVQA samples. AudioRubrics posts the best overall accuracy among similar-size models on MMAU, MMAR, and MMSU, and on MMAU its 78.0% edges out the larger closed Gemini-3.1-Pro at 77.6%.
| Benchmark | AudioRubrics | Strongest similar-size baseline | GRPO (answer-only) |
| MMAU | 78.0 | 77.1 (CESAR) | 75.2 |
| MMAR | 65.8 | 65.3 (Audio-Thinker) | 62.2 |
On MMSU's perception split it beats the best comparable-size baseline by 4.3 points, an 8.9% relative gain. The ablation is clean: dropping the length penalty takes MMAU to 77.2; dropping evolution leaves it at 76.2; dropping static rubrics too leaves 75.2; skipping RL entirely leaves 65.2. Every component contributes.
One finding matters for adopters: the method leans hard on the judge. Swap Gemini-3.1-Pro for the weaker GPT-audio-1.5 as generator and judge, and performance drops below plain GRPO, because imprecise, poorly grounded rubrics become a noisy reward. Response-length dynamics confirm the design: plain GRPO compresses reasoning toward near-zero, removing the penalty makes length explode, and the full method stays in a narrow band.
Most reasoning in audio LLMs today is text priors doing the work; a correct answer does not prove the model listened. This paper pins the training signal to whether the reasoning is actually grounded in the acoustic evidence, and makes the rubric self-updating so it does not saturate. For teams doing audio post-training the recipe is directly portable, and the broader trick, letting reward criteria evolve with the policy, is a useful lever for open-ended RL generally.
The ceiling is set by the judge model. You need a strong audio model that can also write criteria, online, every step, which is expensive and ties results to Gemini-3.1-Pro. Judging and rewriting rubrics for eight rollouts each step adds substantial overhead over vanilla GRPO. Evaluation is multiple-choice audio QA only; whether the grounding transfers to open-ended generation is untested. The authors concede gains scale with judge capability but do not examine what happens once judge and policy converge in ability.