Local Qwen3.8-27B Runs Typed Decisions in <10GB VRAM at 170ms
kyr0x0 · reddit · 2026-09-23
The author released Bonsai-Llama-Jev, a local typed-decision inference system built on Qwen3.8-27B Q264 and llama.cpp, keeping OpenAI API compatibility.
Key numbers
- On typed-decision-bench (175 use cases, 22,000+ decisions), it reaches 76% Soft Accuracy vs 88% for Jev-1.13, ahead of Von-1.1, Laya and Qwen3.5-4B.
- Under 10GB VRAM including KV cache and vision projector; 171ms p50 latency vs 716ms for Jev, roughly 4x faster.
- Lowest Calibration Error (ECE-15) at 13% vs 8.4% for Jev.
Technical points
- Built on a PrismLM llama.cpp fork, implementing TypeSafe AI's System One API directly in the engine via a new POST /v1/systemone endpoint, E2E-tested with official Python/JS SDKs.
- Proposes a calibration.json standard for post-hoc calibration, derived from a math trick found by the author's VAE Auto Research Harness.
- Claims to be the first open Jev-like setup with multimodal typed decisions (images + text/JSON).
- 143 tok/s chat on an RTX 5090.
More from Infra
- French netizen proposes datacenters pay local residents compensation — clemnt · 2026-11-11
- GPU Rental Prices Hit Record Highs: B200 at $6.97/hr, H100 at $3.06 — inductionheads · 2026-11-11
- A Datacenter in Every Office: Building Work-Safe Local AI Racks — dee_hw · 2026-11-11
- "Compute is about power": the AI race's real bottleneck is energy, not chips — _sholtodouglas · 2026-11-11
- CoreWeave CEO says he canceled his 10b5-1 plan, halting insider selling — firstadopter · 2026-11-11
- UK chip start-up Fractile hits $7bn valuation, per FT — evisoft · 2026-11-11