Batching benchmark: 50x throughput for ~24% latency cost on PyTorch
cneuralnetwork · x · 2026-10-23
Benchmarking a small neural net on PyTorch 2.8.0 (MPS), batch size 1→32 kept latency flat at 0.076ms while throughput scaled linearly from 13,178 to 419,863 req/s; at batch 64 latency rose just 24% to 0.094ms for 680,811 req/s. GPUs are built for parallelism: small batches leave compute idle, larger batches buy throughput, not per-request latency.
Related event: PyTorch Benchmarks Show Batching Boosts Throughput at Low Latency Cost(2 posts)→
More from Research
- Dev converts Qwen3-0.6B into a MoE model: quality drops and no speed gain at this scale — Bayliner1980 · 2026-10-23
- Tencent's WeChat Vision releases WeVisOne-VL, unifying object- and pixel-level perception — jacek2023 · 2026-10-23
- De novo designed ligases validate in lab: 27 of 30 computer-designed enzymes work — AllThingsApx · 2026-10-23
- Commenters dissect a "best decision model" leaderboard: same AUC, swing driven by calibration priors — antoine_chaffin · 2026-10-23
- JEPA may keep unpredictable features: new paper gives set-level characterization — hisspikeness · 2026-10-23
- Next-Concept Modeling paper claims 31x faster text generation than diffusion LLM baselines — PanelReader · 2026-10-23