Proposal: audio RefMods for MiniMax-H3, bringing IP-Adapter mechanics to voice timbre
ledadu · reddit · 2026-09-17
The author proposes building "Audio RefMods" for MiniMax-H3, porting the visual RefMod/IP-Adapter mechanism to audio conditioning, and is recruiting collaborators for a POC.
Problem: feeding audio into video/audio DiTs processes it temporally — great for lip-sync, but wasteful on VRAM when you only want global style conditioning (voice timbre, room acoustics, genre), risking OOM.
Mechanism: encode a reference clip with CLAP or MiniMax's native audio VAE; apply attention/average pooling to crush the temporal dimension into 4-16 style tokens; inject via cross-attention through a lightweight adapter into the frozen DiT.
Benefits: stack multiple audio references (voice A + room B) without inference slowdown; the compressed safetensors would be 1-5 MB and load instantly in ComfyUI; style and sync stay decoupled.
Feasibility: training only the adapter on rented A100s for 48-72 hours would cost roughly $100-250. The author seeks feedback from architecture devs, dataset curators and ComfyUI node creators.
More from Multimodal
- Midjourney dual --sref combo produces medieval woodcut cat knight art — sergeantsref · 2026-11-02
- Fluffy AI-generated character designs: fox girl, alien bug and cat photographer — OVolosin82152 · 2026-11-02
- Fable 5.5 reportedly one-shots animation in pure code as Anthropic's coding bet pays off — Hesamation · 2026-11-02
- FlowHMR gets 5x faster, demoing an LLM agent pipeline from video to photoreal mocap — camenduru · 2026-11-02
- Retro 1973 music video "Rock is Dead" by TECHHALLA showcases startlingly realistic AI generation — Mother-Pear-2392 · 2026-11-02
- Weekend splatting with COLMAP and Lichtfeld — janusch_patas · 2026-11-02