New model weight sizes: Q4_K_M GGUF fits a 24GB card with ~20.8GB
AIFlow_ML · x · 2026-10-26
AIFlowML shares weight-size estimates for a newly released open model: BF16 is roughly 66 GB, while Bartowski's GGUF quants come in at 20.8 GB (Q4KM), 25 GB (Q5KM) and 35 GB (Q80) — meaning a single 24 GB card can hold Q4 weights with room left for a short context.
More from Infra
- Reddit asks: Why can't AI coding agents run locally like Stable Diffusion? — Kaporalhart · 2026-10-26
- Vjeux joins Muse Open Kernel Challenge; colinlikescode leads 4 of 5 problems — Vjeux · 2026-10-26
- Gandalf paper: KV cache offloading wears out SSDs in months; attention-aware admission extends life 7.3x — ThePrimeClock · 2026-10-26
- Gary Marcus publicly wagers on whether CoreWeave stock drops below $40 this week — GaryMarcus · 2026-10-26
- Resin: a runtime that lets agents write forward passes to build standalone LLM inference engines — andersonbcdefg · 2026-10-26
- AWS capacity fully booked through 2027 as AI demand surges, CEO says — econoar · 2026-10-26