Metal NVFP4/MXFP8 Kernels Cut Inkling 2 Mini Memory from 46.2GiB to 9.9GiB on M3 Max

GabGarrett · x · 2026-11-22

A notable local-AI improvement for Apple Silicon: Arthur Zucker added native NVFP4 and MXFP8 Metal kernels for Inkling 2 Mini to Hugging Face Transformers, letting its expert weights stay quantized in memory instead of being unpacked.

Result: on an M3 Max, memory usage drops from 46.2 GiB to just 9.9 GiB with identical logits and around 29.6 tokens/sec. This makes Inkling 2 Mini practical to run on 16GB and 24GB MacBooks — real progress for local AI.

Original post →

More from Infra

Infra channel →