RTX 5090 runs a 33B Q4 model at ~86 tokens/s: buy memory bandwidth, not TFLOPS
AIFlow_ML · x · 2026-10-28
A practical local-LLM back-of-envelope: the RTX 5090 has 1,792 GB/s of memory bandwidth, so a Q4-quantized 33B model (21 GB) tops out around 86 tokens/s before context effects. Takeaway: local inference is bandwidth-bound — buy memory speed, not TFLOPS.
More from Infra
- EmbeddingGemma2 Turned Into a 20ms Real-Time Trainable Multimodal Endpoint on CPU — michellechen · 2026-10-28
- Seagate dips after 142% YTD surge, but AI-driven HDD boom keeps margins at records — TiernanRayTech · 2026-10-28
- NetBox Labs partnership targets AI factory config drift, chasing time-to-first-token — ai · 2026-10-28
- US holds 72% of world AI compute as EU's 5% share falls far below its 15% GDP weight — Jsevillamol · 2026-10-28
- Nvidia and Oracle to scale Vera Rubin NVL72 clusters on OCI to multi-gigawatt level — Beth_Kindig · 2026-10-28
- GLM 5.3 Flash NVFP4 on 4x RTX 6000 WS: V2 serving nearly doubles throughput per GPU — TheZachMueller · 2026-10-28