Microsoft Analyzes 13.5M Copilot Sessions: Why Agent Scheduling Differs from Chat
rohanpaul_ai · x · 2026-08-09
A new Microsoft paper analyzing 13.5 million GitHub Copilot production sessions reveals that infrastructure scheduling for coding agents must fundamentally differ from standard chat requests.
- Call Distribution: 87% of LLM calls are initiated autonomously by the agent rather than the user. A single prompt fans out into chains of model calls, tool actions, and retries.
- KV Cache Behavior: Cache hit rates depend heavily on workflow position. Within a single turn, hit rates rise from 45% on the first call to 92–94% by the third. However, it plummets to 55% at turn boundaries and 8% when switching models.
- Idle Times: Median KV-cache idle time is 1.2s within a turn versus 172s across turns. Container idle time jumps from 5.8s to 243s.
- Scheduling Implications: Using turn- and session-level features, the paper's lightweight predictor captures 86–90% of total idle time. Infrastructure should schedule based on workflow state to optimize cache offloading and container reclamation.
More from Infra
- MLCommons launches MLPerf Endpoints, a benchmark that measures inference endpoints the way buyers actually purchase them — TheKanter · 2026-10-16
- Neocloud M&A decoded: CoreWeave, Nebius, Crusoe, Nscale all bet margins vanish beyond GPU scarcity — AccBalanced · 2026-10-16
- AI-Driven Timing-Aware Power Optimization Uses Learned Ranking With Baseline-Preserving Repair — Amit Sharma · 2026-10-16
- Nscale AI datacenter plagued by coolant leaks, transformer fire, 10 safety violations — saranormous · 2026-10-16
- WSL 3.0 ships with native containers and Linux 6.18 kernel, syscall overhead down ~16% — solyarisoftware · 2026-10-16
- Are 22 GB modded RTX 2080 Ti cards a viable budget local LLM option? — Dave86ch · 2026-10-16