Tsinghua Open-Sources Falcon: Visual Target Speaker Extraction With 11x Lower Latency
aigclink · x · 2026-10-31
Tsinghua researchers open-sourced Falcon, an audio-visual target speaker extraction model that isolates a chosen person's voice in real time in noisy multi-speaker environments, and plugs directly into Tencent Meeting and Zoom.
Unlike prior audio-only approaches, Falcon uses visual cues for selective listening: it matches the target person seen by the camera with the microphone signal, then extracts that speech.
Efficiency gains are substantial: vs the AV-TF-GridNet baseline, trainable parameters drop 91.9%, compute falls 20x, and GPU inference latency drops 11x; vs the similarly sized (0.5M param) Swift-Net-12, latency drops 70% and compute 77%.
More from Multimodal
- AI short film made in 12 hours with open-source Longshot Studio, full project files released — R34vspec · 2026-10-31
- Oncologist builds CT reading demo with a 4B open model that boxes lung nodules — Scobleizer · 2026-10-31
- AI film festival Lumara premieres its 74-minute lineup free on X — Scobleizer · 2026-10-31
- PhobosStream: Run 14B Video DiTs on 8GB GPUs With 788MB Peak VRAM via Disk Streaming — martianacademy · 2026-10-31
- 8 AI video models, one prompt: Seedance 2.5 vs Kling 4.0 head-to-head showdown — WorldofAI · 2026-10-31
- Halloween horror video generated on just 8GB VRAM with LTX 2.5 + LoRA — izzray · 2026-10-31