Agent Loops Are SGD Without Gradients: 234 Experiments Cut Inference to 2.32ms/step
jiayq · x · 2026-10-31
The author shares observations from an agent-driven optimization of DeepSeek inference: overnight the loop pushed step time to 2.32ms, below their educated-guess floor of 2.5ms (baseline stock vLLM TP8: 5.654ms). They frame the agent loop as 'stochastic gradient descent without gradients.'
- Gains decay exponentially, not heavy-tailed; agents catch easy wins first, then improvements slow
- Experiment acceptance rate starts high, then stabilizes around 46% (108 of 234)
- Convergence is consistent — agents aren't running out of ideas
- The hard part: discrete optimization over a group of agents with no coordinates, no gradients, and a non-fixed landscape — a sample / reject-or-accept / resample loop
- They suspect Kalman filtering, rejection sampling, or data-driven MCMC as candidate heuristics, though the math is unclear
They see this as engineering and research in one, and plan larger experiments with more trace analysis.
More from Infra
- KV Cache engineering: 1 GiB per 8K-token sequence, where LLM serving performance is won — techNmak · 2026-10-31
- Ollama Cloud has grown 200x since the start of the year, company says — ollama · 2026-10-31
- Datacenter pipeline now totals 94GW vs 82GW in June, with capacity sliding into 2027-28 — bookwormengr · 2026-10-31
- OpenAI bots reportedly hammering domains, fueling speculation of a new training run — gaganghotra_ · 2026-10-31
- AWS details running ChatGPT desktop and Codex via Bedrock in GovCloud with zero data retention — AWS ML Blog · 2026-10-31
- SanDisk Plunges 14% to $1,142 as Poster Rants at KIOXIA Amid Memory Price Surge — zephyr_z9 · 2026-10-31