Tsinghua's Falcon Hits NeurIPS: 0.5M-Param Streaming Lip-Guided Voice Extraction + RealSSA Benchmark
aigclink · x · 2026-10-31
Tsinghua researchers open-sourced Falcon, a streaming audio-visual target speaker extraction framework (NeurIPS 2026 paper), with project page, code, Hugging Face models, and a live demo.
- Method: a single-pass encoder-decoder AudioBlock performs multi-scale time-frequency separation, replacing iterative separator cycles; an encoder-space speaker contrastive loss suppresses near-field leakage at zero inference cost and is architecture-agnostic
- Performance: 0.5M parameters, 51.3ms GPU latency; vs AV-TF-GridNet, parameters drop 91.9%, compute 20.2x, latency 11.1x; SOTA on RealSSA (7.7dB SI-SNRi), LRS2-2Mix, and LRS3-2Mix (16.4dB)
- RealSSA benchmark: a new realistic simulated AVTSE benchmark with 2-6 dynamic speakers across four scene types, plus a real-recorded eval set
- Falcon Studio: runs locally on a laptop, tracks the target's lips via camera, and routes extracted audio to earphones or into Tencent Meeting/Zoom via virtual audio devices
More from Multimodal
- Redditor tests MiniMax cinematography and storytelling in Viking short film — NiceIllustrator · 2026-10-31
- AI-generated fantasy trailer 'The Island of the Banned Gods' shows film-grade visuals — KarimHann · 2026-10-31
- Local video gen on M5 Ultra: Pika H3 Flex + NVIDIA Regenerate-2K, 1088p in 12 minutes — TgoAI · 2026-10-31
- Imagination Studio: open-source AI film studio packs 61+ apps for Mac and NVIDIA DGX Spark — nicolascraske · 2026-10-31
- ComfyUI hairstyle gallery: 800+ references with workflow included — Mysterious_Pride_858 · 2026-10-31
- Hotel room scanned in 4 minutes, Echo-3 reconstructs it perfectly — risingodegua · 2026-10-31