Custom sm_60 kernels push Tesla P100 decode from 11.6 to ~52 tok/s, repo open-sourced
Relative-Ingenuity15 · reddit · 2026-11-23
A Reddit user open-sourced a full set of custom Pascal (sm60) kernels, build flags and llama-swap configs for Tesla P100 16GB. Hand-written GEMVs, XMAD-based NVFP4 expert kernels, faster q80 attention and partial requantization lift Inkling-2-Mini 26B decode from 11.6 to 52 tok/s and prompt processing from 130 to 363 tok/s, with near-lossless quality (98% top-1 agreement, 0.0037 KL). Includes a Qwen3.8-Flash-Next IQ3S setup, per-kernel env-var toggles, and quant selection advice for different VRAM/RAM combos.
More from Infra
- Nvidia expands Vera CPU rack-scale platform: 192 CPUs, 300TB memory, 200TB/s bandwidth — Beth_Kindig · 2026-11-23
- Apple cuts per-dev daily token budget from $300 to $100; cheaper-model strategies follow — Zachly · 2026-11-23
- Open-Source Maestro Video Generator Lures Users Away From $40/Month AI Subscriptions — cocktailpeanut · 2026-11-23
- A 9-machine local AI homelab: 5 agents on one dual-3090 llama-server — StandardLovers · 2026-11-23
- Prune, train, upcycle: one model that scales with your RAM — minhnguyen · 2026-11-23
- Local LLM inference is hopelessly fragmented: one user now juggles 5 engines per model — Fancy-Snow7 · 2026-11-23