Agent bottleneck wasn't latency: token-per-minute ceiling capped product at 2.2 turns/min
Initial_Orange2985 · reddit · 2026-09-05
A developer shares production measurements showing the real ceiling for their agent product was neither latency nor per-call cost, but the provider's token throughput limit: 3600 tokens per turn against an 8000 tokens/min cap, meaning 2.2 turns per minute for the entire product no matter how fast individual calls return.
Key findings:
- A retry loop on 429s feeds the very limit it retries against, making throttling look like an outage. The diagnostic: send a full-history and an empty-history request in the same second — if the empty one fails too, it's throughput, not state.
- Block-by-block token accounting showed the identity/style block was 65.1% of the prompt (median of 30 runs), while user-describing content was just 0.1%. Most of the ceiling went to telling the model who it was.
Open questions posed: when a background queue saturates a token ceiling, drop tasks or queue them? Dropping keeps latency honest but silently loses work; queueing preserves work but turns the limit into an unbounded hang. Also: what prompt cuts survived quality contact, and does routing cheap turns to a smaller model actually help or just move the ceiling?
More from coding & agent
- PostgreSQL 19 slips to November as LLM-assisted code review floods committers with edge cases — sanjaykalra · 2026-10-21
- LangChain adds Human-in-the-Loop beta to Managed Deep Agents — LangChain · 2026-10-21
- Voxtral 3 ships a LiveKit-based voice agent SDK streaming listen-think-speak — MistralAI · 2026-10-21
- Open-sourced pixel art normals generator ships as Rust CLI and wasm library — HugoDuprez · 2026-10-20
- Eval researcher teases guide to subjective-task evals, with a curated resource list — cwolferesearch · 2026-10-20
- Eon cuts engineer onboarding from 2 days to 2 minutes with an MCP Gateway on Bedrock AgentCore — AWS ML Blog · 2026-10-20