LLM inference 101: bandwidth ÷ weight bytes gives your throughput ceiling before any code
abhijithneil · x · 2026-09-04
Engineer abhijithneil started a thread arguing LLM inference is where AI engineers can now get the 'knowing the theory' dopamine rush — and it's a job AI can't easily automate away.
Key points (including replies):
- On every model release, check: BF16 or FP4 quantization? How many params? Smallest deployable machine? How many users can it serve?
- A back-of-envelope formula: N billion weights × 2 bytes each = 2N GB moved per token; divide your GPU's memory bandwidth by that to get the throughput ceiling — knowable before writing any code.
More from Infra
- AMD's Threadripper Halo Station packs 96 cores and 576GB of HBM3E — ccerrato147 · 2026-09-04
- AeroJEPA fluid foundation model joins NVIDIA's PhysicsNeMo ecosystem — ricardovinuesa · 2026-09-04
- Building a €2-2.5k local AI rig for legal RAG and agentic coding: hardware picks debated — whatyathinkk · 2026-09-04
- Dual 3090 owners debate adding more cards: bigger local models vs parallel instances — Blues520 · 2026-09-04
- NousResearch brings one-click local model setup to Hermes Agent on NVIDIA systems — lifebypixels · 2026-09-04
- Regulated-industry dev seeks AI Gateway with Okta SSO and runtime policy enforcement — IrrepressibleInk · 2026-09-04