Falcon: Streaming Audio-Visual Speaker Extraction Cuts Params 91.9% and Latency 11×
_akhaliq · x · 2026-10-30
Falcon is a causal streaming audio-visual target speaker extraction model that uses the target speaker's mouth video to isolate their voice from noisy, overlapping speech. It pairs a single-pass multi-scale time-frequency separator with an encoder-space speaker contrastive loss that cuts interfering-speaker leakage without extra inference cost.
The paper also introduces RealSSA, a benchmark spanning varied speaker counts, near/far-field interference, reverberation, and sudden noise, with simulated scenes plus a held-out real-recorded evaluation set. Against causal AV-TFGridNet, Falcon reportedly cuts trainable parameters by 91.9%, compute by 20.2×, and GPU inference latency by 11.1×. Code, weights, datasets, and interactive demos are available on the project page.
Related event: Tsinghua's Falcon Extracts Target Speaker with 91.9% Fewer Parameters(2 posts)→
More from Multimodal
- User says Higgsfield wiped her paid subscription and denied it; lists alternatives — aziz4ai · 2026-10-30
- Comfy Cinematic: open-source mobile browser client for ComfyUI with auto workflows — No-Sleep56 · 2026-10-30
- Viral bobblehead-hugging-giant-Snickers image prompt delivers cute results in seconds — aitrendz_xyz · 2026-10-30
- Spark Radix launches: a WebGPU Gaussian splat renderer built on three.js — theworldlabs · 2026-10-30
- AI-generated slapstick comedy short "D*ckhead" debuts — Scobleizer · 2026-10-30
- Phonic-1 handles a 44-minute Brazilian hotel call with 123 tool calls — graceisford · 2026-10-30