LEGATO 2: Toward Multimodal Sheet Music Recognition and Understanding
Guang Yang, Brian Siyuan Zheng, Victoria Ebert, Noah A. Smith
cs.CV, cs.AI
2026-07-07
UW's Legato 2 segments systems with YOLO and transcribes each with a 113.7M VLM, including embedded text. On camera string quartets, OMR-NED falls from Legato 1's 58.2 to 31.6.
Optical music recognition has been stuck between pipelined modules that pass errors forward and end-to-end models that either read one system or treat the page as one blob, ignoring how scores are actually read left to right, system by system. Embedded text (titles, composers, inline marks) is a blind spot for neural OMR; Legato 1 replaced every text span with a <|text|> token.
Frontier vision-language models are weak on sheet-music questions too. On MusiXQA's hard split, Gemini 3 Flash scores 8.4 and GPT 5 Mini 8.0 with no transcription. This paper treats OMR as a context provider for those models rather than training another generalist score reader.
Three stages. YOLOv8 medium (26M) crops musical systems from each page, fine-tuned on 1,024 hand-boxed pages, half from the synthetic PDMX-Synth train split and half from real IMSLP engravings. A 113.7M autoregressive VLM then transcribes one system at a time, conditioned on the current crop and up to 1,024 tokens of prior system-level ABC, inside a 2,048-token window. A rule-based converter merges those fragments into standard voice-wise ABC.
System-level ABC is the alignment trick: a system's encoding does not depend on voices that appear only later, so training and inference can follow reading order. The tokenizer adds byte fallback on BPE at vocabulary 4,096, so rare characters survive as bytes instead of being erased. Lyrics stay out of the corpus; the authors treat them as loosely coupled to structure and leave them to off-the-shelf OCR.
Downstream understanding is a prompt change: feed Legato 2's transcription with the image to GPT-5 or Gemini 3.1 Pro and let the frontier model use the symbols.
Page-level OMR-NED (lower is better):
| Dataset | Audiveris | Gemini 3.1 Pro | Legato 1 | Legato 2 |
| PDMX-Synth test, 411 pp. | 56.3 | 90.3 | 28.6 | 23.5 |
| Rendered string quartets, 252 | 64.6 | 93.5 | 32.9 | 17.1 |
| Camera string quartets, 252 | 75.4 | 94.1 | 58.2 | 31.6 |
| IMSLP piano, 32 | 71.5 | 89.1 | 44.3 | 34.2 |
Dense scores move most: camera quartets drop from 58.2 to 31.6. Ablations show system segmentation alone cuts validation OMR-NED from 70.2 to 48.5; byte fallback mainly buys text; 4,096 is chosen on validation.
On camera quartets, total text CER is 73.8 for Audiveris, 58.1 for Gemini, 24.8 for Legato 2. Title CER is 10.6. Unordered CER versus PaddleOCR is 25.1 against 85.5. Composer names stay bad because page numbers get labeled as composers.
On understanding, Gemini 3 Flash on MusiXQA goes 8.4 with no context, 20.1 with Legato 1, 25.3 with Legato 2. Gemini 3.1 Pro on SSMR-Bench goes 71.4 to 84.8 to 92.7.
The practical recipe is specific: do not force a general VLM to read staves. Run a system-sliced OMR model and pass ABC as context. A 113.7M specialist plus a frontier VLM is cheaper and more accurate than stuffing the page into Gemini. Code and data are promised after publication.
The authors flag two gaps. They did not quantify how YOLO misses pollute downstream VLM reasoning. Training is fully synthetic, so engraving style, scan quality, and handwriting still hurt; camera Lieder is 43.6 versus Legato 1's 44.9, almost a tie. Lyrics are out by design. Composer CER sits near 100. Absolute OMR-NED is not clean transcription either: 31.6 is still about a third of an edit distance.