Napkin math for GPU fit: a 7B model needs ~14GB at FP16, ~3.5GB at INT4
cneuralnetwork · x · 2026-10-28
@cneuralnetwork shares a quick rule for estimating whether a model fits on your GPU: weight memory ≈ parameters × bits / 8. A 7B model needs roughly 14GB at FP16, 7GB at INT8, and 3.5GB at INT4. But weights aren't the whole story — KV cache, activations, and runtime overhead (which depend on context length and batch size) also consume VRAM, so fitting in memory doesn't mean comfortable inference. Always leave headroom.
More from Infra
- CXMT to start small-scale HBM3E production within two months, yield at 20% — zephyr_z9 · 2026-10-28
- Video Transcription Costs Under 1 Yuan per Hour, CPU Free but Error-Prone — huangyun_122 · 2026-10-28
- A forgotten supercomputer design: bare dies, gold post interconnects, 3D stacking, immersion cooling — jwt0625 · 2026-10-28
- Reddit predicts a GPU price crash next year as the compute bubble deflates — sayamss · 2026-10-28
- Karpathy recommends a YouTube series building a CPU from a single transistor — blaizedsouza · 2026-10-28
- Idle GPUs burn cash: Neolabs lose money below ~60% spot utilization — FinanceYF5 · 2026-10-28