Multi-Task Multi-Frame Visual Piano Transcription
Yonghyun Kim, Hoyeol Sohn, Juhan Nam, Alexander Lerch
cs.SD, cs.AI, cs.CV, cs.MM, eess.IV
2026-08-04
V2N is the first complete video-only piano transcription system, predicting onset, offset, key hold, and velocity from 1 s of video and setting SOTA on PianoVAM and R3.
Automatic music transcription (AMT) recovers each note's pitch, onset, offset, and velocity from a signal. Audio-based piano transcription is mature on pitch, onset, and velocity, but offset is confounded by the sustain pedal: once the pedal is down, the key has returned to rest while the sound keeps ringing, so audio systems align note offsets to pedal release instead of physical key release, drifting seconds away from the MIDI NoteOff event.
Visual piano transcription (VPT) watches the keyboard directly. A pressed key looks different from a released one regardless of pedal state, so the physical key state is directly observable. It also holds up when audio is ambiguous, degraded, or absent (multi-instrument mixes, noisy or reverberant rooms, footage with a missing audio track). The trouble is that VPT has been neglected: existing methods use at most 0.2 s of video context, supervise only the center frame of the window, leave offset accuracy far behind onset (PPAN's offset F1 at 50 ms on PianoVAM is just 45.9%), and have never reported note-level velocity. V2N (Video to Notes) sets out to close those gaps.
V2N has three stages: a visual feature extractor, a shared temporal backbone, and four task heads.
The visual front end follows S2S: a ResNet-18 trained from scratch on grayscale input, with a learned "slope prior," a 1D spatial encoding that maps each horizontal position to one of the 88 keys. Each frame uses a 5-frame (0.2 s) window, global-average-pools, and projects linearly into the backbone dimension. The backbone is three Conformer ConvModule blocks (FFN, depthwise 1D convolution with kernel 31, FFN; self-attention dropped). Three kernel-31 convolutions give every output frame a receptive field over all 25 frames of the 1-second input, without the quadratic cost of self-attention. The four heads are each a bidirectional LSTM (hidden 256) plus a linear projection, outputting onset, offset, key hold, and velocity.
The four targets are grounded in key mechanics: onset is a key press; offset is physical key release, aligned with MIDI NoteOff; key hold marks the key being held over [onset, offset); velocity is per-key normalized MIDI velocity. The first three are sigmoid-bound and binarized at 0.5 at inference; the velocity head is linear, clipped to [0, 1] and rescaled to integers 0 to 127.
Two choices matter. First, per-frame supervision: labels are placed on every frame of the 1-second clip, not only the window center, and training segments are densely sampled at 1-frame stride and shuffled across videos, so every frame index serves as a center at some point. That yields about 5x the supervised signal per segment at no inference cost. Second, offset-guided note decoding: a note starts at an onset peak and ends at an offset peak, with key hold as a fallback if it drops first. This differs from the audio convention of cutting notes by a frame-activity threshold alone, and the ablation shows it is more accurate.
The model costs 125 GFLOPs and 28.2M parameters per 1-second clip (the visual front end is 99% of that); with a 0.5 s overlapping stride the amortized cost is about 250 GFLOPs per second, roughly 40 ms of GPU time on an RTX 5080, real-time factor about 0.04.
The main experiments run on PianoVAM (107 top-down amateur-practice videos with synchronized MIDI from a Yamaha Disklavier) and R3 (31 hours of practice recordings by two professional pianists; harder, with multiple pianos, camera positions, lighting, and advanced repertoire).
On PianoVAM, V2N matches Li et al. on onset F1 (94.7 at 50 ms, 97.0 at 100 ms, within 0.5 percentage points) and beats every baseline on every offset-dependent metric. The biggest gains are on physical key release:
| Metric (50 ms) | PPAN | V2N |
| +Off (with offset) | 45.9 | 89.5 |
| +Off+Vel (offset + velocity) | 29.7 | 78.3 |
Dedicated offset supervision with 1 s of context nearly doubles offset F1 and lifts +Off+Vel by 48.6 points over PPAN. V2N is the only system accurate across all four MIDI attributes (pitch, onset, offset, velocity), and the first to report note-level velocity F1 from video.
On R3, V2N beats prior VPT on onset in both splits, and the offset gap widens: onset-centric baselines collapse on offset metrics, confirming that physical key release needs dedicated supervision rather than emerging as a by-product of key-hold modeling.
The ablations mirror the title. On task heads, removing a head hurts onset and collapses offset F1; the heads give complementary supervision and inference cues. On frames and architecture (each row adds one component): center-frame loss extended to all 5 frames gives +1.3 onset and +4.5 offset; adding the Conformer sequence model gives +2.5 onset and +5.7 offset, the single largest jump, because a purely spatial front end cannot tell pressed from held; stretching the window from 0.2 s to 1.0 s adds only 0.4 and 0.8 points on PianoVAM but is decisive on R3.
For anyone doing temporal event detection or multimodal transcription, this is a clean engineering reference: the visual side can recover the physical key-release information that audio loses to the pedal, and the multi-task, multi-frame supervision plus offset-guided decoding are reusable design choices. It is also about 1.7x cheaper than Li et al.'s video branch, and the 1-second window carries no real wall-clock penalty.
Be honest about scope, though. This is a narrow, clearly incremental engineering contribution, not a methods-level breakthrough. Its value is pushing VPT from rough onset-only output to complete MIDI, and nailing every design choice with an ablation.
The authors name two main limits. First, cross-dataset transfer collapses: the learned pixel-to-key mapping does not generalize across camera geometries, even after perspective-transform normalization (median keyboard width is about 606 px on PianoVAM versus 784 px on R3, so the same column indexes a different key). Second, V2N does not estimate sustain-pedal control changes.
Two more concerns stand out. R3 has synchronization issues of its own: 70 of 895 files have audio-video offsets above 200 ms. The authors keep the original test split because it hits all models equally, but excluding the 10 affected R3x test files lifts V2N's 100 ms onset F1 from 86.6 to 94.3, a 7.7-point jump, so the reported R3 numbers are suppressed by sync noise. Also, every baseline is retrained by the authors; although their S2S and V2R onset reproductions land within about 1 point of the originals, the offset baselines were weak to begin with, and retraining differences could still widen or narrow the gap.