Kinetic, a Rust inference engine, answers questions on a 22.6k-token doc in 0.13s vs llama.cpp's 10.95s
jasonkneen · x · 2026-11-12
The new open-source Rust inference engine Kinetic showed striking speed on modest local hardware (RTX 3060 6GB laptop GPU + Ryzen 5 6600H): after pasting a 22,631-token document, llama.cpp took 10.95s while Kinetic answered in 0.13s. The key trick is that Kinetic starts processing while the user is still typing, moving work into time already spent. A separate file-rewriting benchmark is also notable. Developer Jason Kneen has opened an issue and PR adding macOS Apple Silicon support: running on an M3 Max CPU at 5 tok/s, with NEON quantized kernels (Q4K–Q80), f16/bf16 dot products, CLI updates, CI matrices — 27 files, 1,500 lines, though the roadmap originally had no macOS plans.
More from coding & agent
- Local vs Cloud AI Coding Agents: Benchmarks, Hardware Costs, Performance — luisdans · 2026-11-12
- The simplest heuristic for agent UX: expect it to behave like a capable coworker — andreisavu · 2026-11-12
- Agent Substrate 1.0 GA: microVM-sandboxed agent runtime with zero ambient credentials — rakyll · 2026-11-12
- Dev builds sketch-guessing app with new Gemini Decisions API, guesses terrible drawings — fofrAI · 2026-11-12
- Together Compiler: agent-driven continual compilation for AI inference — togethercompute · 2026-11-12
- Book Review: Building AI Agents for Finance — a practical multi-agent patterns reference — bibryam · 2026-11-12