Lucebox's bespoke inference engine beats llama.cpp with 95.4 tok/s on Qwen 3.8 27B
QuixiAI · x · 2026-11-13
Lucebox built its own inference engine tuned for one specific machine (RTX 495 + R9700), shipping on every box with day-0 model support and weekly optimizations. Decode benchmarks: Qwen 3.8 27B at 95.4 tok/s vs 64.8 on llama.cpp; Qwen 3.8 Flash-Next at 81.5 vs 53.9 on Strata. On 120K prefill it also leads (1,621 tok/s vs Strata's 1,362). The team's thesis: hardware-software co-design for one machine beats generic optimization.
More from Infra
- SGLang-Neuron v0.1.0 open-sourced: SGLang serving stack now runs on AWS Trainium — ying11231 · 2026-11-13
- Nvidia Has $36B in Off-Balance-Sheet Backstops for Neoclouds, Plus $20B in Leases — SumitGup · 2026-11-13
- India's pretraining gap isn't GPU smuggling — it's underinvestment, argues researcher — teortaxesTex · 2026-11-13
- Report: Jensen Huang resigns as Nvidia CEO amid -63% cash conversion, $47B CoreWeave debt — SumitGup · 2026-11-13
- Intel partners with RadixArk to optimize SGLang for Intel CPUs and GPUs upstream — ying11231 · 2026-11-13
- Microsoft, AWS, Google and more hit by simultaneous mass outage — zainhas · 2026-11-13