Indie dev open-sources Leviathan, a 3.5-year self-funded local LLM inference suite with hand-written CUDA kernels
The-world-is-a-stage · reddit · 2026-10-25
A solo developer released Leviathan on GitHub after 3.5 years of self-funded work: a local LLM inference toolkit where models run directly on your GPU with no server in between — no llama.cpp, no LM Studio.
Highlights:
- Hand-written CUDA kernels covering every quantization and full-precision variant
- 30+ architectures he built over time merged into one program
- A custom merge tool replacing mergekit, with a 4-way mode that previews merge outcomes before committing
- Roots in a 2023 persistent-memory bridge for VS Code, later shipped as a .vsix extension
Free to download, with a large README; a Hugging Face Space is also available.
More from Infra
- ByteDance, Alibaba and Tencent order 3M+ Nvidia RTX PRO 5500 cards worth $35B+ — Miles_Brundage · 2026-10-25
- Nebius hits 67B tokens/day on OpenRouter, but rivals process up to 2T daily — AccBalanced · 2026-10-25
- OpenRouter analysis finds evidence of spoofing-style price manipulation by inference providers — AccBalanced · 2026-10-25
- EmbeddedLLM receives AMD MI350X server donation to advance vLLM ROCm work — AccBalanced · 2026-10-25
- 106 Tesla Megapacks to replace a former Belgian coal plant with 400 MWh storage — XFreeze · 2026-10-25
- Running Qwen 3.8 Flash Next on 2x CMP 170HX mining GPUs, config shared — leonbollerup · 2026-10-25