Dev ships C-based local inference engine targeting tool calling on low-VRAM hardware
ZenZombie117 · reddit · 2026-10-04
A developer released Xyntetik-Runner 1.0.0, a self-built local inference engine written in C, aimed at agentic workflows on constrained hardware rather than raw speed versus llama.cpp:
- Fixed truncated tool calling: responses return valid JSON directly, no parse-retry loops
- Schema enforcement to eliminate invalid tokens — the author calls this a killer pain point for agent workflows
- No idle memory usage: RAM is only consumed when the model is actually called
- LoRA training directly on 4-bit files, no FP16 copy needed
- Token-identical to llama.cpp, verified token-for-token on Gemma4-Moe
- Supports Metal, CUDA, and Intel/AMD CPUs, plus an OpenAI-compatible --serve endpoint
The project began on an 8GB M1 Mac and a 3070 with 8GB VRAM; access to a 24GB MIG slice of a friend's Blackwell card expanded the scope. Runner will stay free forever, with future tracks targeting enterprise-grade verified local inference and research (the earlier Xyntetik-Kvist-14B model grew out of this work).
More from coding & agent
- Claude Code overtakes Cursor and Copilot in job-post mentions ahead of Anthropic IPO — antgoldbloom · 2026-11-19
- Dev submits PR adding periodic checkpoints to GKE sandboxes so agents don't lose progress — yoimnotkesku · 2026-11-19
- Orchestrator's Cloud Agents graduate from beta to general availability — julianweisser · 2026-11-19
- Zero users, 33 pages shipped: dev crowdsources audit for AI-agent trust schema — TheHamzaShahbaz · 2026-11-19
- She vibecoded her own photobooth with Claude Code — and shared the prompt — toolstelegraph · 2026-11-19
- Should AI Agents Write to Your Memory or Only Read? Dev Weighs In — jordimr · 2026-11-19