1740 TPS claim challenged: speculators trained on benchmark data and P90 metrics inflate inference numbers
VoidAsuka · x · 2026-11-15
A reproduction found that the 1740 TPS decode on 4xB300 only holds on the showcased prompt with an unusually high acceptance rate (5.6 tokens/step). On typical coding workloads with thinking enabled it drops to 1200 TPS, and 750 TPS on chat/writing. Commenter caoshuxun notes widespread "benchmaxxing" in inference: training speculators on benchmark datasets to game numbers, and criticizes the use of P90 latency metrics that don't reflect real user-perceived speed.
More from Infra
- Why No One Can Replicate Jensen Huang's Asian GPU Supply Chain — _onionesque · 2026-11-15
- Running 3 AI agents in parallel: even 32GB machines hit memory limits — PawelHuryn · 2026-11-15
- Curated 10-resource path to learn LLM distributed training from scratch — East-Muffin-6472 · 2026-11-15
- Report: Nvidia's interim CEO reviews AI customer financing, $300B OpenAI Ohio chip deal in doubt — SumitGup · 2026-11-15
- Skip the $3,000 Mac mini: a $1,000 renewed Dell micro PC works fine for agents — PawelHuryn · 2026-11-15
- Pirate Face mirrors open models as torrents, already indexing 28,890 magnet links — zainhas · 2026-11-15