PianoBench: open audio models top out at 52.8% on musical reasoning

kastnerkyle · x · 2026-11-05

PianoBench: testing musical perception and reasoning in LLMs

Researchers (including Chris Donahue) introduce PianoBench, a benchmark requiring language models to reason about piano performances across congruent audio and symbolic input representations. It derives from a new training-scale dataset: 20,000+ hours of interleaved transcripts of music and spoken commentary from instructional piano videos.

Key findings:

Open-weight audio-language models reach at most 52.8% accuracy, only modestly above a no-audio control;
Frontier models with chain of thought reach 93.2% when given music transcripts as text, exposing a gap between perceiving musical features from audio and reasoning over explicitly stated musical events;
Mid-training on interleaved musical instruction raises Qwen 3.5 9B Base's zero-shot accuracy from 42.8% to 56.0% without chain of thought.

The authors suggest instructional videos may be an overlooked source of multimodal supervision for music.

Original post →

PianoBench: open audio models top out at 52.8% on musical reasoning

More from Multimodal