Strix Halo int4×int4 WMMA: Gemma 4 31B at 2.43× llama.cpp with matched quality

No-Dragonfly6246 · reddit · 2026-11-17

A Reddit user reports enabling the int4×int4 WMMA instruction on AMD Strix Halo's 8060S (measured 104 TOPS, 2× the FP16 rate) that neither ROCm nor llama.cpp uses. They wrote a custom GEMM kernel and run Gemma 4 with int4 weights and int4 activations, preserving accuracy via rotation + GPTQ + selective mixed precision.

Performance (llama-bench, 31B QAT Q40, 2k prompts, single stream):

Quality (full test sets, 31B): GSM8K 96.4 vs 96.2, IFEval 91.7 vs 92.2, MMLU-Pro 85.1 vs 85.1 — essentially matched.

One-command install via embedl (embedl serve google/gemma-4-31B-it/ryzen-ai-max-395). This fills a real gap: AMD iGPU matrix cores that mainstream inference stacks ignore, highly relevant for local-deployment enthusiasts.

Original post →

More from Infra

Infra channel →