Custom engine runs Inkling 2 Mini at 129.5 tok/s with 344K context on a single RTX 3090, 2.3x faster than llama.cpp
teortaxesTex · x · 2026-11-21
Developer superalesha open-sourced maya-ik2, a Project Maya-based inference engine that pushes Inkling 2 Mini to 129.5 tok/s decode with 344K context on a single RTX 3090 — 2.3x faster decode and 3.7x faster prefill than llama.cpp with zero quality loss.
Key techniques:
- All 3,072 experts kept resident in VRAM; kernel launches per token cut from 7,350 to 781 via a single CUDA graph (over half of llama.cpp's decode time went to 1-2 µs glue ops)
- Custom Tensor Core kernels for FP8 KV attention (776 GB/s) and NVFP4 experts — the 3090 has no FP4 hardware, so 4-bit weights are unpacked to FP16 in registers on the fly, feeding FP16 Tensor Cores at 745 GB/s
- BF16 vision moved to CPU (4.5s new image, 0.5s cached) freed 1.3 GB VRAM for the 344K context
The repo is MIT-licensed and supports images and tools.
More from Infra
- Researcher builds local GPU server to run 24/7 AI agent system after Mac mini runs out of memory — nodosenlared · 2026-11-21
- Counterintuitive: KV cache quantization in llama.cpp uses MORE VRAM than f16 — Fancy-Snow7 · 2026-11-21
- Cohere launches confidential computing with encrypted inference; exec says rivals' ZDR labels are marketing — Nils_Reimers · 2026-11-21
- Gary Marcus shares chart of Oracle CDS spiking parabolically to record highs — GaryMarcus · 2026-11-21
- Can TEEs and Remote Attestation Make Decentralized GPUs Safe for Sensitive AI Workloads? — MrSwannn · 2026-11-21
- AI gateways are becoming the key control layer in production AI stacks: 8 capabilities — goyalshaliniuk · 2026-11-21