Qwen3.8-27B benchmark: 6 inference engines, GGUF vs NInfer, 256K context speed AND quality
fuzhongkai · reddit · 2026-11-06
A comprehensive benchmark runs Qwen3.8-27B across six inference engines, measuring not just TTFT/prefill/decode throughput and concurrency but also output quality, across 64K/128K/256K contexts and both GGUF and NInfer formats. Key findings: rankings diverge sharply by metric; long-context performance reshuffles the leaderboard, with strong short-prompt configs falling behind at 128K–256K; and quality validation should be standard in inference benchmarking. The author discloses he wrote TensorSharp, one of the engines compared.
More from Infra
- Jane Street shares its experience building with NVIDIA's cuTile language — GuggerSylvain · 2026-11-06
- AI overbuild skeptics miss the demand: agents, cyber AI and 0.1% enterprise penetration — builderjaydub · 2026-11-06
- Against datacenter doom: three reasons the buildout continues — max_paperclips · 2026-11-06
- iQuantize: open-source desktop workspace for guided LLM quantization to GGUF, MLX and ONNX — Zipidyzip · 2026-11-06
- Developer ships server-side barge-in for local LLM voice assistant VoiceManager — HockeyDadNinja · 2026-11-06
- LSEG cuts 5-year market backtest from hours to ~1 minute; Tampa General adds AI safety gate — NVIDIAAI · 2026-11-06