Do cold-start requests belong in local model benchmarks? The often-ignored first call
Greedy-Badger-8463 · reddit · 2026-10-01
The author flags a blind spot in local model benchmarking: warm numbers look good, but the first request after restarting the app takes forever. He suggests interactive-app benchmarks should report cold-start alongside warm numbers, and asks what people actually measure — both matter, just for different reasons.
More from Infra
- Huawei Mate 90 debuts Kirin 9050 Pro, first chip with LogicFolding architecture — ingliguori · 2026-10-01
- NVIDIA A20 Standard Reportedly Skips WMCM Packaging — 'Not Enough Capacity' — zephyr_z9 · 2026-10-01
- TRL hits 1M post-trainings per month, team eyes 1M per week — QGallouedec · 2026-10-01
- oMLX 0.7.0 lands with up to 50% faster long-context decoding and a rebuilt memory guard — pcuenq · 2026-10-01
- SlideDP fine-tunes Qwen2.5-72B on four RTX 4090s, beating FSDP2 throughput by 11.2% — Ruijia Yang · 2026-10-01
- China sets all-time monthly electricity record, but industrial power demand growth slows — teortaxesTex · 2026-10-01