Open-source pipeline turns a song into a beat-locked music video on one RTX 5080
Adebrantes · reddit · 2026-10-29
The author released on-the-one (MIT), a build-competition entry for #OpenSourceHiggsfield. The piece "Green Flag" has 44 shots and goes from song to music video where every cut lands on the beat, running fully locally with no API nodes and no credits.
The pipeline has six stages:
- Planner: lyric sheet, Whisper word timings and a beat grid become a shot list; cuts sit on bar lines and are pinned to the words being sung
- Animatic: a full-length animatic over the real song serves as storyboard, with per-shot direction written while it plays
- Look and cast: lock an art style and build character sheets once; every shot pulls characters from those references so the chorus dog matches the verse dog
- Three rolls: three keyframe options per shot (FLUX Klein); the pick is animated by LTX audio-to-video against that shot's exact slice of the song
- Foley: LTX generates sound effects, picked by ear and mixed under the song
- Finish: 1080p upscale then a frame-exact conform over the original audio
Everything runs on one RTX 5080; shipped workflows have all API nodes stripped, with a build check enforcing it. The author asks what users would want next: different models per step, a lighter GPU tier, etc.
More from Multimodal
- Foyege Draconia AI portrait shot wows Reddit with photoreal detail — nobodyreadusernames · 2026-10-29
- Mistral Large 4 'Le Chonk' lands on Hugging Face: 1T-param MoE, 49B active, 1M context — ariG23498 · 2026-10-29
- Self-hosted realtime avatar server: 30 FPS to 90 FPS on a free Colab T4 — SavingsWeather1659 · 2026-10-29
- Runway launches Character Swap: replace video characters while keeping motion intact — runwayml · 2026-10-29
- A 30-second AI animated short 'The Letter Came By Owl' made with WAN 3.0 — Burak_Antakya · 2026-10-29
- Prompt turns static charts into documentary-grade animated data visualizations in minutes — TawohAwa · 2026-10-29