GLM-5.3-Flash Q4 Hits 37.4 t/s at 300k Context on M3 Ultra via Custom Kernels
IngeniousIdiocy · reddit · 2026-09-09
A developer deep-tuned GLM-5.3-Flash (Q4) local inference on M3 Ultra, open-sourcing the work in the ds4 repo's glm53-m3ultra branch:
- Decode: fused dozens of small kernels into larger dispatches, raising memory-bandwidth utilization from 59% to 81%; 29→40 t/s short-context, 24→38 t/s at 62k.
- Long-context attention: replaced staged sort/merge of candidate lists with parallel scans, byte-identical outputs, end-to-end 300k context 21.6→37.4 t/s.
- Prefill: batching weight reuse and wider loads took 366→550 t/s (+50%); cold 62k processing dropped from 170s to 113s.
- Speculative drafter: windowed admission controller backs off when not paying; +4% on a 32-request agent session, +20–50% on SQL/JSON, 1% cost on prose.
- Quality: NLL essentially unchanged (0.300766 vs 0.300804) and 90/100 first-token matches on the 100-prompt QA set.
The optimizations are M3 Ultra-specific, based on detailed measurements of its two-die memory behavior, system cache, and Metal scheduling.
More from Infra
- Cloud capacity crisis is now an AI architecture problem, argues new newsletter — DavidLinthicum · 2026-10-26
- SpaceX files $40B GPU financing commitment with Apollo, already at 1.7GW — XFreeze · 2026-10-26
- Meta plans to raise up to $95B to fund massive AI expansion — Polymarket · 2026-10-26
- Nebius CEO: Debt Markets Closed for Neoclouds Last Week, Leaning on Customer Advances — zephyr_z9 · 2026-10-26
- CoreWeave 5-year CDS spikes to ~1,100bps as market fears AI bubble starts as credit event — kimmonismus · 2026-10-26
- AI demand reshapes cloud computing: scarce capacity and long-term deals replace on-demand — DavidLinthicum · 2026-10-26