Rust weight-streaming engine runs FLUX.2 9B on a 12GB RTX 3060, ~9% faster with 2GB resident pool
madtune22 · reddit · 2026-09-23
Reddit user madtune22 released StreamLoader, a Rust engine that streams GPU weights from SafeTensors via mmap, pinned host memory, reusable VRAM buffers and CUDA-event prefetching, exposing tensors to PyTorch through DLPack. Tested with FLUX.2-klein-9B on an RTX 3060 12GB; a ResidentPool keeping 2GB resident improved generation time 9% vs full streaming. A Qwen-Image-2.1 port worked without Rust changes. Code is open for feedback on prefetch design, residency strategies, and stress-test architectures.
More from Infra
- Cloudflare CTO Dane Knecht makes TIME's exec list, rebuilding web for AI agents — michellechen · 2026-09-23
- New Qwen3.8 27B quants beat ISTA and Unsloth byte-for-byte on three corpora — Dutchnamn · 2026-09-23
- crabbox now runs on boxd: isolated KVM microVMs with ms boot times for repo commands — steipete · 2026-09-23
- AI data centers now demand up to 1,000 MW, a 200x jump in power — Olivier__OG · 2026-09-23
- AMD MI355X Hits 1.7x Perf-Per-Dollar vs DGX B300 via SGLang KV Cache Fix — zephyr_z9 · 2026-09-23
- Apsara Conference's real story: China's full AI stack, chips to robotics — manishkhosiya · 2026-09-23