Coding Agent Optimizes Its Own Serving: Kimi K3 Throughput Up 43% via InferenceForge
rohanpaul_ai · x · 2026-11-14
The Agent Economics team released InferenceForge: a recursive self-improvement loop where an off-the-shelf coding agent does SLA-aware inference optimization over a custom environment, harness, and eval suite. The agent measures cache hit rates and per-kernel timing, writes its own microbenchmarks, then lands targeted fixes.
Results: under a 30 tok/s per-user latency SLA, it pushed SGLang-based Kimi K3 serving to 12.8k tok/s per GPU — 43% above the best published AgentX reference (8.9k) — in 9 iterations over 3 days, with every fix landed as a PR on an SGLang fork.
Biggest wins came from agent-traffic-specific fixes (a few very long contexts beside many short requests):
- Forcing at least 8 splits per request in the attention decode kernel: previously split count depended only on batch size, so at 40+ concurrent requests each ran on a single thread block and the longest context set every decode step's time (417 µs per launch under AgentX load vs 55 µs with uniform lengths).
- Two more mixed-traffic fixes detailed in the blog post, along with lessons learned and all code changes.
More from coding & agent
- Gensyn launches evidence-mining agent contest with $7,250 prize pool — benfielding · 2026-11-14
- 'Peak prompt engineering': full prompt for having an LLM draw a JS-only masterpiece — adonis_singh · 2026-11-14
- Superconductor adds Loops MCP connector so coding agents can manage email campaigns — sergeykarayev · 2026-11-14
- Dev records 2 hours of manual Neovim coding to get PRs accepted — yacineMTB · 2026-11-14
- Grok Bot tests auto-approving purchases with $50-$5,000 spend limits — XFreeze · 2026-11-14
- Cloudflare: AI pentesting framework traffic surged 30x; a six-pillar "Modern Security Model" to fight back — Cloudflare Blog · 2026-11-14