Running big models on a single RTX 4090: INT8 weights, FP8 decoder, RAM tiering
AIFlow_ML · x · 2026-10-25
AIFlowML summarizes a practical setup for running large models on a single RTX 4090: INT8 weights and cache, FP8 decoder, each module moved to GPU only during its stage, with 18 latent history frames kept on GPU and the rest archived in system RAM. Key takeaway: budget for RAM, not just VRAM.
Sources included: the model card (see 'Model Assets' for the INT8/LoRA line) is worth reading in full, with code and demo linked in the post — a concrete reference for local video-model deployment on consumer cards.
Related event: Running LLM and Video Generation Inference on a Single RTX 4090(2 posts)→
More from Infra
- NVIDIA shows off an 8-unit DGX Spark cluster at GTC Berlin — MaziyarPanahi · 2026-10-25
- 32GB VRAM cards spike overnight: B70 jumps from $1.4K to $2K, derailing local LLM builds — tableball35 · 2026-10-25
- 'The moat is how much energy you have': AI competition boils down to power — airesearch12 · 2026-10-25
- ByteDance, Alibaba and Tencent order 3M+ Nvidia RTX PRO 5500 cards worth $35B+ — Miles_Brundage · 2026-10-25
- Nebius hits 67B tokens/day on OpenRouter, but rivals process up to 2T daily — AccBalanced · 2026-10-25
- OpenRouter analysis finds evidence of spoofing-style price manipulation by inference providers — AccBalanced · 2026-10-25