Local Qwen3.8-27B Runs Typed Decisions in <10GB VRAM at 170ms
kyr0x0 · reddit · 2026-09-23
The author released Bonsai-Llama-Jev, a local typed-decision inference system built on Qwen3.8-27B Q264 and llama.cpp, keeping OpenAI API compatibility.
Key numbers
- On typed-decision-bench (175 use cases, 22,000+ decisions), it reaches 76% Soft Accuracy vs 88% for Jev-1.13, ahead of Von-1.1, Laya and Qwen3.5-4B.
- Under 10GB VRAM including KV cache and vision projector; 171ms p50 latency vs 716ms for Jev, roughly 4x faster.
- Lowest Calibration Error (ECE-15) at 13% vs 8.4% for Jev.
Technical points
- Built on a PrismLM llama.cpp fork, implementing TypeSafe AI's System One API directly in the engine via a new POST /v1/systemone endpoint, E2E-tested with official Python/JS SDKs.
- Proposes a calibration.json standard for post-hoc calibration, derived from a math trick found by the author's VAE Auto Research Harness.
- Claims to be the first open Jev-like setup with multimodal typed decisions (images + text/JSON).
- 143 tok/s chat on an RTX 5090.
More from Infra
- Intelligence may become abundant before power plants and fabs can keep up, argues thread — r0ck3t23 · 2026-09-23
- Cloudflare CTO Dane Knecht makes TIME's exec list, rebuilding web for AI agents — michellechen · 2026-09-23
- Parakeet.java ports NVIDIA's Parakeet ASR to pure Java, transcribing 15-min speech in ~15s — mukel90 · 2026-09-23
- Nvidia-backed Nscale omits its biggest customer Bytedance from US IPO filing — The Decoder · 2026-09-23
- New Qwen3.8 27B quants beat ISTA and Unsloth byte-for-byte on three corpora — Dutchnamn · 2026-09-23
- crabbox now runs on boxd: isolated KVM microVMs with ms boot times for repo commands — steipete · 2026-09-23