Same weights, dumber model: 100k-token logit study exposes inference stack pitfalls
量子位 · wechat · 2026-08-29
Level1Techs user thr3e ran rigorous experiments capturing full logits across 100k+ tokens with Qwen3.6-27B on an RTX PRO 6000 Blackwell, explaining why local LLMs often feel dumber than official ones.
- Attention backends: swapping only vLLM's FlashAttention2 / FlashInference / TritonAttention caused Top-1 flips in long contexts; one traced case corrupted a tool call (wrong Cisco interface), cascading into two more failed calls. Same-backend reruns were bit-identical, so divergence comes purely from different CUDA kernels.
- KV cache quant: BF16 stayed stable; INT8 recovered after errors; INT4 flip rates spiked in long context and tool calls failed irrecoverably.
- Weight quant shootout: a community INT8 (W8A16, no calibration) beat Qwen's official FP8 and NVIDIA's NVFP4; NVFP4 was worst, nearing 50% Top-1 flips at 88k context and failing real tool calls.
- Tensor parallelism quirks: identical BF16 weights succeeded at TP1, failed at TP2, succeeded again at TP4 — traced to NCCL cross-card reduction numerics.
The vLLM nightly container carries 734 packages; the author warns that low KL-divergence numbers on model cards are uninterpretable without full disclosure. Test tooling is being packaged for distribution.
More from Infra
- A look at running a small local AI cluster — BLUECOW009 · 2026-08-29
- Moore Threads MTT S5000 achieves Day-0 support for Zhipu GLM-5.3-Flash — teortaxesTex · 2026-08-29
- Lightning AI adds H200s and launches VMs in early access — LightningAI · 2026-08-29
- Blogger pushes back: no US data center has ever cut household water access — AndyMasley · 2026-08-29
- Voters flip on data center bans once projects cover grid, water costs — Polymarket · 2026-08-29
- AI Data Centers Turn to Fuel Cells to Bypass Multi-Year Grid Delays — tengyanAI · 2026-08-29