Pure-C Inference Engine Runs 35B MoE Models on a 2014 Tesla K80 at ~21 tok/s
Electrical_Ninja3805 · reddit · 2026-10-21
A developer released Pantheon, a dependency-free C inference engine (built on the Foundry ML library) that targets Kepler GPUs, letting a cheap used Tesla K80 from 2014 run 35B-class MoE models.
Measured on one K80 (both dies) in a 2-core, 8 GB, no-AVX host:
- Qwen3.6-35B-A3B and Nex-N2-mini (Q3KM) decode at 21 tok/s
- Prompt reading at 110-120 tok/s up to 8k tokens, 75 at 14k, 55 at 30k
- Weights live in VRAM, so an 8 GB host suffices
Built-in prefix caching cuts opencode's second turn (14k-token system prompt) from 170 s to 9 s. The main weakness is prefill speed — pasting a 30k-token codebase means minutes before the first token. Ships with a one-line installer, an OpenAI-compatible API, and also runs Ornith 1.5 on Tenstorrent Blackhole cards.
More from Infra
- Cadence announces sovereign AI cloud for European chip designers — karlfreund · 2026-10-23
- SGLang Deep-Dive Blog Series Breaks Down KV Cache, Posts up to 15.5x Throughput Gains — BanghuaZ · 2026-10-23
- DynaLM: Open-Source CPU-First LLM Inference Engine Seeks Feedback — Unlikely_Safety_7456 · 2026-10-23
- Capital Is Now the AI Buildout Bottleneck, Says Analyst Citing Jensen Huang — JOBhakdi · 2026-10-23
- DynaLM: an open-source CPU-first LLM inference engine seeks testers and feedback — Unlikely_Safety_7456 · 2026-10-23
- Cerebras explains how it deploys trillion-parameter models on wafer-scale chips — bookwormengr · 2026-10-23