Strix Halo int4×int4 WMMA: Gemma 4 31B at 2.43× llama.cpp with matched quality
No-Dragonfly6246 · reddit · 2026-11-17
A Reddit user reports enabling the int4×int4 WMMA instruction on AMD Strix Halo's 8060S (measured 104 TOPS, 2× the FP16 rate) that neither ROCm nor llama.cpp uses. They wrote a custom GEMM kernel and run Gemma 4 with int4 weights and int4 activations, preserving accuracy via rotation + GPTQ + selective mixed precision.
Performance (llama-bench, 31B QAT Q40, 2k prompts, single stream):
- Prefill pp512: 679 vs llama.cpp 287; pp2048: 724 vs 276
- Decode with MTP: 24.4 vs 11.0 tok/s
- End-to-end: 31B 2.43×, 12B 2.80×, E4B 1.58×, E2B 1.39× (31B: 15.3 vs 6.3 tok/s)
Quality (full test sets, 31B): GSM8K 96.4 vs 96.2, IFEval 91.7 vs 92.2, MMLU-Pro 85.1 vs 85.1 — essentially matched.
One-command install via embedl (embedl serve google/gemma-4-31B-it/ryzen-ai-max-395). This fills a real gap: AMD iGPU matrix cores that mainstream inference stacks ignore, highly relevant for local-deployment enthusiasts.
More from Infra
- How rollout staleness hurts on-policy distillation, and how to build a faster pipeline — burkov · 2026-11-17
- SK hynix's Nvidia revenue doubled to $24B, but US customer concentration is the real risk — tengyanAI · 2026-11-17
- macOS TCP Stack Breaks After Exactly 49.7 Days of Uptime Due to 32-bit Timestamp Overflow — DanielLockyer · 2026-11-17
- Running a 2M-param embedding model on an ESP32 with no_std Rust at 830 embeddings/s — qdrant_engine · 2026-11-17
- Voice agent failed 0/5 turns in background noise; adaptive client-side VAD gate fixed it at lower latency — shrapknife · 2026-11-17
- Hierarchos-Native v0.2.0: Rust+Vulkan LLM training adds RDNA3 WMMA and native multimodal training — PhysicsDisastrous462 · 2026-11-17