Microsoft details Maia 200: no cache hierarchy, software-defined dataflow at 10K TFLOP/s FP4
beenwrekt · x · 2026-08-27
Microsoft published the detailed architecture of Maia 200 (arXiv:2608.24664), its second-gen inference accelerator, already in production in the Azure fleet and built for trillion-parameter frontier models.
The headline decision: no cache hierarchy. The paper argues LLM inference is mostly data-oblivious — compilers can plan nearly every memory access before the model runs, and caches exist to guess what data comes next. When the schedule is known, guessing hardware only adds cost: tag arrays and remapping logic incur 30-35% area and energy overhead plus 10-15% extra access latency.
Maia 200 replaces guessing with orchestration: an architecture called SDLA (Software Defined Locally Accessed Dataflow) exposes every scratchpad, DMA engine, semaphore, and compute unit to software, with C++ control programs running ahead of the data path. Spec-wise, the chip delivers 10,145 TFLOP/s FP4 and 5,072 TFLOP/s FP8 within a 750W TDP, with 7 TB/s HBM bandwidth. This shift from thread-centric to data-movement-centric design yields significant cost and energy savings.
Related event: Microsoft Details Maia 200 Accelerator, Ditches Cache Hierarchy(3 posts)→
More from Infra
- How Many Users Can One DGX Spark Realistically Serve? Community Asks for Numbers — edge_compute_user · 2026-08-27
- Unsloth requested to re-quantize older Qwen models using UD 3.0 — Fancy-Snow7 · 2026-08-27
- QNX partners with Hailo for edge Physical AI: 14x performance consistency — pdamodaran · 2026-08-27
- US Holds 15-20x Compute Advantage, But May Not Matter for Some Threats — ohlennart · 2026-08-27
- Hark partners with NVIDIA for gigawatt-scale compute on Vera Rubin platforms — adcock_brett · 2026-08-27
- Minimax H3 Local Benchmark: 5-Second Clip Takes 4 Minutes on AMD 7900XT — thevictor390 · 2026-08-27