Running Nemotron Omni Natively in Pure MLX on Mac
divinetribe1 · reddit · 2026-08-07
Nvidia's open-weights Nemotron Omni model supports vision and audio, but the existing 4-bit MLX quantization only loaded the text backbone on Mac. To fix this, the author rewrote the vision and audio towers, processor, and token splicing in pure MLX.
- Implementation: Ported directly from Nvidia's PyTorch reference; runs the LLM in 4-bit quantization while keeping the towers in bf16.
- Accuracy: Tested every component against the PyTorch reference with identical inputs and weights. All 23 tests passed, showing near-perfect cosine similarity (approx 1.0) for both audio and vision.
- Performance: On an M5 Max, it achieves 67.7 tok/s with images, 147 tok/s with audio, and 152 tok/s for text-only. Peak memory on the image path is 22.1GB, fitting comfortably on a 32GB Mac. The code is MIT licensed and open-sourced.
More from Infra
- SpaceX to Invest $16.8B in Terafab Targeting 1 Terawatt Compute Demand — ChrisGPT · 2026-08-07
- Profiling is the Highest Leverage Skill in High-Performance Engineering — Abhishekcur · 2026-08-07
- TSMC Shows ASML EUV Machines in Action at Arizona Fab — saibharadwaj · 2026-08-07
- AI Drives Spending Surge for Top 3 Hyperscalers as Neoclouds Emerge — DavidLinthicum · 2026-08-07
- Running MiniMax H3 Video Generation on RTX 5070 Ti Incurs 240s Overhead — orlandogourmet66 · 2026-08-07
- AWS Introduces Rate Limiting for Bedrock AgentCore Gateway — AWS ML Blog · 2026-08-07