Learned video codecs break across devices. Microsoft MLVC fixes cross-platform decoding at 100 FPS.

MLVC: Multi-platform Learned Video Codec for Real-World Deployment

Tanel Pärnamaa, Martin Lumiste, Ardi Loot, Evgenii Indenbom, Andrei Znobishchev, Ando Saabas

ECCV 2026

eess.IV, cs.AI, cs.CV, cs.LG

2026-06-26

MLVC encodes scale parameters into the bitstream so decoding stops breaking across devices, hitting ~100 FPS on commodity NPUs and beating hardware HEVC by 70%+ in subjective quality.

What problem this solves

Neural video codecs swap HEVC-style hand-tuned modules for neural networks and now beat classical codecs on compression. But they have never been deployable, for two reasons: heavy compute, and a more fundamental cross-platform problem.

Applications like video conferencing run the encoder and decoder on different devices, often chips from different vendors. The paper's opening failure says it plainly: encode a clip on an Apple M3 NPU, decode it on an Intel NPU, and the strongest existing neural codec, DCVC-RT, produces completely corrupted frames.

The root cause sits in entropy coding. The codec needs a scale parameter describing the latent distribution to run entropy decoding. Each side computes it on its own floating-point hardware, and the tiny differences accumulate over the prediction chain until decoding blows up. The obvious fix, quantizing to INT8, does not work either. Apple's Neural Engine simulates INT8 in FP16, and every vendor's compiler makes different choices, so bit-exact parity is off the table.

Method

MLVC's central move is to stop computing the scale and send it instead. The scale parameters are compressed 128x through structured parameter sharing, packed into the hyperprior, and transmitted in the bitstream. Both sides read the scale from the same stream, so entropy decoding agrees without bit-exact arithmetic. The cost is a higher bitrate.

Four changes claw the efficiency back:

• Gated memory. DCVC-RT uses recurrent features as temporal state, which has limited capacity for occluded objects. MLVC adds an explicit memory with forget and output gates that selectively retains information across longer horizons.
• ReGLU activation. Activations are highly sensitive, and NPUs compute them through vendor-specific piecewise approximations, which is itself a source of divergence. MLVC sticks to ReLU-family activations and uses ReGLU gating for expressiveness, with ReLU1 capping values at 1 to stabilize training. The gating adds no measurable runtime cost.
• Long-term reference (LTR) frames. Borrowed from traditional codecs, these periodic reference frames shorten the prediction chain against float drift and improve packet-loss robustness. They also let the intra period grow from 64 to 1024, saving more bitrate.
• Perceptual training. Video conferencing cares most about faces, so MLVC weights an LPIPS loss with a face-segmentation ROI mask. It deliberately skips GAN loss to avoid temporal flickering.

Results

Benchmarked on the VCD video conferencing set against hardware HEVC-QSV, the strongest deployable classical baseline:

DCVC-RT is genuinely stronger same-platform, but it collapses cross-platform and is therefore undeployable. MLVC sits about 18 percentage points behind DCVC-RT same-platform, the direct price of the cross-platform constraints. Among neural codecs that actually decode across platforms, MLVC leads by a wide margin: -45.2% versus +15.0% for the prior fixed-prior approach on HEVC B.

On speed, both encode and decode average over 100 FPS across the three NPUs: Apple M3 Pro does 129.5 encode / 122.9 decode at 360p, Intel Lunar Lake 98/98, Qualcomm Snapdragon X 80/77.5. In the cross-platform matrix, most encoder-decoder pairings land within 2 BD-rate points. The larger MLVC-S model reaches 1080p at 30 FPS while keeping -37% BD-rate.

Why it matters

This is the first neural video codec that is cross-platform robust, real-time on commodity NPUs, and competitive on compression at the same time. For end-to-end, cross-vendor workloads like video conferencing, neural codecs have until now been theoretical. Microsoft open-sourced it (github.com/microsoft/mlvc) and tuned it explicitly for commodity NPUs rather than high-end GPUs, which signals a deployment-first intent. The PSNR gap versus DCVC-RT same-platform is real, but it buys a codec that actually runs, and that trade is the right one for shipping product.

Limitations

The authors name their own: power consumption becomes the next frontier once real-time is solved; simultaneous encode and decode at high resolution (1080p and above) on a single device is still hard for symmetric architectures; the unified I/P-frame model trades BD-rate for deployment simplicity; and the regular I-frames that cross-platform operation requires are overhead that single-platform setups would not need. One more caveat: the subjective-quality comparison is anchored almost entirely on HEVC-QSV. Stronger classical codecs like VTM and ECM were omitted because they lack commodity hardware implementations, so the "strongest deployable baseline" framing should be read with that scope in mind.

Terms

• Neural video codec: a codec that uses neural networks for inter-frame prediction and entropy modeling, aiming for fewer bits at equal quality.
• BD-rate: the relative bitrate change at fixed quality, negative meaning it beats the baseline; the standard efficiency metric for codecs.
• Hyperprior: a side channel in neural codecs that models the latent distribution and is transmitted separately to the decoder.
• Entropy coding: the final lossless compression step driven by symbol probabilities; if the two sides disagree on the probabilities, decoding breaks.
• NPU: a neural processing unit built into phones and laptops for inference, such as Apple's Neural Engine, Intel Lunar Lake, or Qualcomm Hexagon.

Source

• arXiv 2606.28027v2
• PDF
• Hugging Face

What people are saying

• Microsoft's MLVC tackles cross-platform bottleneck in neural video codecs tanelai

Related papers

• Samsung's KNN tonescale thresholds beat U-Net and NILUT at adaptive color grading
• One DISCO operator covers 9-to-72-view CT without retraining, +3.4 dB over CNNs
• Pathology ViTs need rotation aug: PathDino holds 0.85 m-kNN, Virchow only 0.53
• When Is Underwater Color Restoration Solvable? Haifa Team Proves Error Vanishes with Camera Resolution
• One one-hot pattern embedding lets a single network demosaic Single-, Quad-, and Nona-Bayer sensors

All paper explainers