Full local AI music-video pipeline revealed: YuE2, Wan2.2, Z-Image and ffmpeg in ComfyUI
EnvironmentLocal8149 · reddit · 2026-10-22
A Reddit user shared their end-to-end ComfyUI workflow for AI music videos, from song to finished cut, and asked for advice (the pain points: realistic shots, hands, and multi-person scenes falling apart):
- Song: local YuE2 generation, with audio.cpp for vocal separation, speech recognition and alignment
- Cast portraits: Z-Image Turbo text-to-image (1920×1088)
- Keyframes: Qwen-Image-Edit 2511 + Lightning LoRA for character images, then Z-Image Turbo img2img at denoise 0.3 for skin texture
- Singing shots: Wan2.2-S2V, keyframe + 3s vocal stem through a wav2vec2 audio encoder for lip-synced clips
- Dance/story shots: Wan2.2-I2V two-stage hybrid (high-noise then low-noise model with speed-up LoRA)
- Smoothing: FILM frame interpolation ×3 (16→48fps)
- Edit/finish: ffmpeg beat cuts, 24fps blend, grade, grain, 1080p
- Thumbnails: Qwen-Image-Edit in the video's style
They also note Claude-assisted iteration eventually made things worse and had to be reverted, and wonder what tools high-quality parody videos use.
More from Multimodal
- Customuse launches Smart Mesh AI with geometry-aware adaptive character clothing tool — SimplyAnnisa · 2026-10-23
- AI-generated movie poster 'The Chase' crams a wolf, a plane and muscle cars into one chase — Aggravating-Sir-115 · 2026-10-23
- SteerTTS-edge: an 11.84MB steerable TTS with unchanged WER running 19x realtime on CPU — Working_Hat5120 · 2026-10-23
- ElevenLabs' Chimera pairs Eleven v4 voice with Seedance video to make character images speak — AIwithGhotai · 2026-10-23
- Open-Source Tennis Models Plus Claude Turn Video Into a World-Class Coach — tweetsatpreet · 2026-10-23
- Berkeley's StyleStream 2.0 cuts streaming voice conversion latency to 520ms, 2x faster — Berkeley · 2026-10-23