Gemma 4 31B QAT on one 24GB card: 46 tok/s plain, ~80 with draft model on RTX 4090
Subject_Poet_3861 · reddit · 2026-10-16
Glyd 0.31 runs Google's Gemma 4 31B and 12B QAT checkpoints on a single 24GB card using its own engine.
Quantization: none of their own — they store Google's QAT format exactly (4-bit codes plus a scale per 32 weights, 4.5 bits/weight), bit-for-bit identical after decode, verifiable via glyd verify against sha256s (lossless vs the QAT checkpoint, not the original bf16).
Speed (single user, greedy, 256 tokens): 31B on RTX 4090 at 46 tok/s plain, 78-80 with Google's draft model (up to 176 on predictable text); 26 tok/s on RTX 3090; 12B on RTX 4080 SUPER at 82.5/109 tok/s. Draft-model greedy output is token-identical. 30K-token prompt: 20s on 4090; 32K context uses 21.8/24.6 GB.
Quality vs QAT weights: wikitext-2 mean KLD 0.018, median 0.003, top-1 96% (differences stem from bf16 summation order); GSM8K 72/75, MMLU-Pro 70/75.
Limits: Linux + RTX 30 series+, text only, one request at a time; 26B-A4B/E4B/E2B unsupported yet. Next: lossless bf16 checkpoint support (31B in 38 GiB instead of 58).
More from Infra
- Meta's open Muse Spark needs 493GB of memory — far beyond any consumer laptop — TejasKumar_ · 2026-10-16
- CoreWeave plunges 20% as credit markets price >60% default odds; is this dot-com all over again? — kimmonismus · 2026-10-16
- MiniCPM-V 4.7 35B-A3B hits 17+ tokens/s on iPad with ~3GB memory — techNmak · 2026-10-16
- China rolls out 'token factories' and a 2026-28 token economy action plan — nordicinst · 2026-10-16
- Vercel Sandbox Drives go GA: attach TBs of data in sub-second, fork instantly — cramforce · 2026-10-16
- NVIDIA launches RTX Spark for local AI on Windows PCs — ostrisai · 2026-10-16