Inkling-2-Mini GPTQ Quant Ships 524K-Token Context in One 18.3GB File on a Single 24GB Card
W61k3r · reddit · 2026-11-23
Wa1k3r released Inkling-2-Mini-8Q-RCO-GPTQ on Hugging Face: a single 18.3 GB file that runs a 524,288-token context on one 24 GB GPU, with the model's own draft head bundled inside for speculative decoding. The file was produced by LexiPanel's quantization tool and runs on LexiEngine, a Rust inference engine — part of a self-built local inference stack (LexiSuite) designed around this model.
More from Infra
- Anthropic's on-site power is 80% simple-cycle gas; health damages estimated at $2-3B per year — petrusenko_max · 2026-11-23
- Agent-run OpenRouter routing eval: cheapest providers are unreliable, fastest nearly as cheap with caching — MikePFrank · 2026-11-23
- Dev: 99% of people shouldn't self-host local LLMs, and he only does it for the hardware — gnukeith · 2026-11-23
- Samsung Labs open-sources LittleBit, compressing LLM weights to 0.1 bits — MikePFrank · 2026-11-23
- Tech Sells Data Centers as Local Wins, But Jobs Are Just Guards and Remote Engineers — feregri_no · 2026-11-23
- RSIBench-Infra from ByteDance Seed and Tsinghua tests if agents can optimize production-scale AI infrastructure — geoffwolfe · 2026-11-23