Running Nemotron 75B on M2 Max

These_Meaning_3883 · reddit · 2026-07-12

The author successfully ran Nemotron Puzzle 75B on a 64GB M2 Max and added native nemotronhpuzzle support to mlx-lm.

Benchmark Results

Comparing 4-bit and 5-bit expert quantization under identical prompts, 5 random seeds, and temp 1.0 / topp 0.95:

4-bit experts: checkpoint 42.03 GiB, peak memory 49.68 GB, average 14.27 tok/s, passed 24/30 tasks
5-bit experts: checkpoint 49.88 GiB, peak memory 58.12 GB, average 10.53 tok/s, passed 21/30 tasks
Long-context retrieval: 4-bit scored 4/5, while 5-bit scored 0/5

Key Takeaways

The author suspects that on a 64GB machine, 5-bit quantization operates too close to the memory limit. The growth of the KV cache and working set degrades both performance and stability.
Results for 5-bit might differ on 96GB or 128GB machines.

Technical Details of the Adaptation

Added blockwise config and tensor remapping for Nemotron-H, matching NVIDIA's FP32 norm/router behavior.
Encountered an issue where the first layer SSM output had a cosine similarity of only 0.8832 with the reference implementation.
Root cause: NVIDIA computes softplus(dt + dtbias) in BF16 before casting to FP32, whereas mlx-lm's shared path casts to FP32 first. Fixing this boosted similarity to 0.999998.
BF16 output heads are mandatory: quantizing the lmhead for a 131k vocabulary to 4-bit produces repetitive garbage outputs.

More from Infra