M5 Macs can go faster: w8a8 kernels lift Gemma4 prefill from 2,193 to 3,029 tps
maddie-lovelace · reddit · 2026-07-24
A Reddit user says Apple M5 silicon is not yet being fully used by current inference backends.
- MLX and llama.cpp still run 16-bit activations everywhere, even though M5 reportedly supports INT8 activations / w4a8 dtype.
- The author built w8a8 kernels and reports 1.4x faster Gemma4 prefill performance.
- On an M5 MacBook Air, prefill throughput rose from 2,193 tps to 3,029 tps on a 130,173-token input.
- For shorter contexts, the author says it gets even faster and approaches 10k tps.
More from Infra
- Michael Dell: Musk is "the most demanding customer there is" — Colossus servers built in ~6 weeks — elonmusk · 2026-10-27
- AI chip startup Tenstorrent raises $1.5B at $6.2B valuation, more than doubling — Beth_Kindig · 2026-10-27
- Anthropic discloses up to $120.5B SpaceX compute deal, ~$3.1B monthly at full capacity — XFreeze · 2026-10-27
- Satire: swapping Raft for an LLM-style decision model tanks write throughput 99.98% — irombie · 2026-10-27
- Cadence joins the domain-specific AI model camp for chip design — karlfreund · 2026-10-27
- Qdrant builds Universal I/O to run vector search mmap code on object storage — qdrant_engine · 2026-10-27