Moondream Launches Photon Inference Compiler, Throughput Up to 2.33x Faster

Moondream introduced Photon, an inference compiler designed to compile model descriptions into an optimized megakernel. This allows the entire inference process to run directly on the GPU, eliminating CPU-to-GPU overhead and freeing up CPU resources. Official benchmarks show significant throughput improvements on the H100 compared to vLLM and SGLM, signaling that inference is entering the compiler era.

Confirmed

• Photon 2.0 is positioned as a compiler-driven inference engine, supporting the compilation of models like Moondream, Qwen 3.5, and Gemma 4 into megakernels.
• The core mechanism compresses the entire forward pass into a single GPU program, allowing the GPU to execute the full inference chain directly.
• Official benchmarks covering 1, 2, 4, and 8 concurrent requests show throughput improvements ranging from 1.01× to 2.33× on the H100 compared to vLLM and SGLang.

Why it matters

• Moondream believes this approach signals that AI inference is entering the compiler era, removing the need for developers to manually tune for every deployment scenario.
• By automatically generating optimized megakernels, this technology aims to significantly reduce scheduling overhead and boost hardware execution efficiency.

2026-08-04 ~ 2026-08-04 · 6 related posts

• Photon 2.0 compiles Moondream, Qwen 3.5 and Gemma 4 into megakernels — sloppenheimer · 2026-08-04
• [source] Moondream pitches an inference compiler that serves 1.01×–2.33× vLLM and SGLang throughput — suchenzang · 2026-08-04
• Moondream says its inference compiler emits one megakernel for the whole model — AccBalanced · 2026-08-04
• Photon claims up to 2.33× throughput over vLLM and SGLang on H100 — AccBalanced · 2026-08-04
• Moondream argues inference is entering a compiler era with GPU megakernels — AccBalanced · 2026-08-04
• Photon 2.0 targets physical AI inference and claims 2.3x throughput over vLLM — MikeBirdTech · 2026-08-04