NTU's MIDI-REPA uses MIDI alignment to improve text-to-music, trainable on one RTX 3090 in ~31h
affige_yang · x · 2026-10-28
A National Taiwan University team introduced MIDI-REPA, a representation-alignment framework that improves text-to-music generation with symbolic supervision.
Core idea: melody, rhythm, harmony and structure can be compactly represented as MIDI, yet current models learn them only from dense audio. MIDI-REPA derives MIDI supervision via automatic music transcription (AMT) and aligns the intermediate representations of a flow-matching DiT.
Key contributions:
- AMTFlow: a transcription pipeline combining an instrument tagger and MuScriptor
- JamendoMIDI: multi-instrument MIDI transcriptions of 51,077 instrumental Jamendo tracks (CC BY-NC-SA 4.0)
- Built on the 122M-parameter FluxAudio-S baseline: aligning a MuScriptor-derived piano roll at an early DiT layer across the full flow trajectory works best
- With no extra data or techniques, it stays competitive on ATTM objective metrics while training on a single RTX 3090 in about 31 hours
Paper, code, dataset and demos are open-sourced; the authors argue symbolic supervision is a simple, cheap learning signal for text-to-music generation.
More from Research
- Johns Hopkins lands up to $52M from ARPA-H to build AI that decodes protein misfolding — JosephJacks_ · 2026-10-28
- Microsoft's Diagnostic AI published in Science and open-sourced, retested on Gemini 4 Argon — vivnat · 2026-10-28
- Machenta robot reacts and recovers in real time, trained 100% in simulation — Ed__Johns · 2026-10-28
- DeFM Depth-First Foundation Model: Why Anchor to Depth, Not Estimator Features — chris_j_paxton · 2026-10-28
- AI-powered formal verification finds real Redis/Valkey bugs, incl. a 23-char freeze — LingmingZhang · 2026-10-28
- Variant effect predictors are shaping the ClinVar labels used to train and evaluate them — pastramimachine · 2026-10-28