Helping Music Co-Creation Agents 'Listen' Well: Hierarchical Self-Supervised World Models for Understanding and Generation
Scott H. Hawley
NeurIPS 2026 Creative AI Track
cs.SD, cs.LG, eess.AS
2026-08-05
In a 2.55M self-supervised model, rhythm and phrase emerge but harmony needs supervision; a chord head lifts unsupervised key detection from 0.16 to 0.70.
A collaborative music-creation agent should listen to what you played, offer verbal feedback and a few musical suggestions, and never just do the work for you. The author borrows a metaphor: an AI Rick Rubin, the producer who says he "barely" plays an instrument but is decisive about taste. The bottleneck is not the brain: an LLM can already discuss music. It is the ears, turning music into representations rich enough to drive both understanding and generation. Large audio-language models underperform smaller targeted models on simple music tasks, and prior perception-plus-generation systems (including the author's own diffusion-based Pictures of MIDI) were slow enough to need a GPU, locking out musicians without good hardware. This paper builds a self-supervised model small enough to answer in seconds on a CPU, handling both listening and suggesting.
The system is a hierarchical self-supervised "world model" standing in for the ears and mouth of a larger agent. The encoder is a 2.55M-parameter Swin V2 trained self-supervised on MIDI piano-roll images, reusing the earlier MIDI-RAE-JEPA objectives: pitch and time equivariance, SIGReg against collapse, masked embedding prediction, and a soft factorization loss. Inputs are 128x128 binary piano-roll crops, the time axis at 32nd-note resolution, four bars per window, with velocity deliberately discarded to suit a songwriting rather than performance setting.
One constraint drives the design: inference has to be timely on a CPU. That fixes the crop at 128x128 and pushed generation from diffusion to flow matching, which has straighter trajectories and needs only a few Euler steps whereas diffusion inpainting needs many. The flow runs in pixel space (no latent compression), conditioned on per-level embeddings reduced by PCA (at least 90% variance kept, a 3x compression). A single trick does double duty: the multi-level conditioning dropout used in training both controls how far a generated variant strays from the source and enables graphical inpainting. The user paints a region on the roll, the conditioning there is zeroed, and the model fills from surrounding context, with no inpainting-specific sampler.
The probing experiments yield the central finding: the hierarchy level at which a musical property becomes linearly decodable tracks the time scale it operates on. Phrase boundaries read best off the coarsest levels (L0 to L2), note density and harmonic detail off the finest (L4, L5).
More striking is what emerges for free versus what has to be asked for. Temporal and phrase structure appear from the self-supervised objectives alone; harmonic content barely shows up. Adding a small chord-supervision head lifts root identification from 0.25 to 0.59, joint chord from 0.18 to 0.54, and key detection, which is never supervised, from 0.16 to 0.70. The cost is small: phrase-boundary AP slips from 0.27 to 0.25 and temporal offset from 0.26 to 0.24.
The DINOv2 baseline is instructive: on properties that reduce to image statistics it is competitive or better (note density 0.93, sharper cross-song separation), but it falls away on music-specific probes (chroma 0.36 versus 0.65, phrase AP 0.21 versus 0.27). The authors are explicit that these harmony scores read as capacity probes, not downstream performance; dedicated chord detectors do far better.
Generation: conditioned on a real window's own embeddings, the flow reproduces it at pixel F1 0.996. In inpainting, restored note density falls monotonically with dropout strength (76% with fine levels fully dropped, 53% with all levels dropped).
Efficiency: encoding a window takes 8.6 ms on two CPU threads, and a 10-step Euler suggestion takes 2.8 s on a full CPU, 0.6 s on Apple MPS, and 0.10 s on a laptop RTX 4090. A live demo ships with the paper.
For anyone building music tools or creative agents, three takeaways. First, it is a serious attempt at a world model for symbolic music: no music-theory vocabulary, no labels, and rhythm and phrase structure still emerge, suggesting self-supervision works on structured symbolic signals and not only on natural images. Second, the CPU-speed constraint forced a clean set of choices (fixed 128x128 window, pixel-space flow matching, multi-level conditioning dropout) that let conditional generation and graphical inpainting share one pipeline and one training run, removing the need for a separate repair sampler. Third, the practical reminder: do not expect self-supervision to surface everything. Harmony, which needs explicit semantics, is cheaper to ask for with a little supervision than to wait for, and the jump from 0.18 to 0.54 on chords and 0.16 to 0.70 on unsupervised key detection makes the case.
The authors are candid that harmony scores are capacity probes, not usable performance; for real chord recognition, use a dedicated small model. The binary roll discards velocity, multiple instruments, and note onsets, compressing polyphonic performance into "is there a note," which caps expressive headroom. The window is fixed at 128x128 (four bars), so long-range structure can only be extended later by treating the coarsest levels as tokens in a sequence model, still untried. Data is still dominated by POP909 pop songs; on the larger Lakh set, a 4x subset gave no consistent gain at matched training budget and even degraded chroma and cross-song separation, suggesting the recipe is not very data-hungry but also that its ceiling may be untested. Inpainted notes "sometimes land off," admitted to track surrounding material less tightly than larger diffusion models, excused by the idea that an agent only needs to suggest an idea for the human to realize. Finally, this is a single-author, small-scale study; the key "emerges versus asked-for" claim rests on probe interpretation and has not been validated in a real end-to-end collaborative songwriting workflow.