Gemma.c: full Gemma 1/2/3 multimodal inference in a single C file, ~72 tok/s on CPU at INT8
This-Purchase-3325 · reddit · 2026-11-16
- Gemma.c is a from-scratch inference engine for Gemma 1/2/3 (including multimodal models) written in a single C file, needing only libc and optional OpenMP, with fp32/fp16/bf16 and int8 (W8A8) weights.
- Multimodal prompts via @image{} and @longimage{} (Gemma 3 pan & scan); image decoding uses an embedded minified stbimage.
- Benchmarks (Core Ultra 5 225H, WSL2, 10 threads): gemma-3-1b-it INT8 decodes 72 tok/s, prefills 570 tok/s (up to 850 tok/s for 1000-token inputs); the 4B vision model hits 20 tok/s. INT8 is 3.5x faster than fp32 and 2x faster than fp16.
- Known limitations: no W8A8 on Gemma 1, slow bf16 without AVX512-BF16, native Windows 1.7x slower than WSL, no batching. Pre-exported INT8 models are on HF, no Python/torch needed.
More from Infra
- Sarvam AI Launches Token Factory: India-Hosted Inference for Open Models — itsOmSarraf_ · 2026-11-16
- BIS Issues Interim Final Rule Addressing Remote Access to Advanced AI Chips — apples_jimmy · 2026-11-16
- Jailbroken PS4 runs a 9B open-source model at up to 43 tok/s by talking directly to its 2013-era GPU — AIFlow_ML · 2026-11-16
- Mobile AI Factory: agents auto-build on-device SLMs with 19x less GPU time — pmttyji · 2026-11-16
- vLLM serves DeepSeek V4.1-Flash at 954K tokens/s per GPU, $38.96/GPU-hour on GB300 NVL72 — bookwormengr · 2026-11-16
- Colossus 2's 1.2M B200s could generate ~$400B/year in tokens at DeepSeek Flash rates — bookwormengr · 2026-11-16