Agent bottleneck wasn't latency: token-per-minute ceiling capped product at 2.2 turns/min
Initial_Orange2985 · reddit · 2026-09-05
A developer shares production measurements showing the real ceiling for their agent product was neither latency nor per-call cost, but the provider's token throughput limit: 3600 tokens per turn against an 8000 tokens/min cap, meaning 2.2 turns per minute for the entire product no matter how fast individual calls return.
Key findings:
- A retry loop on 429s feeds the very limit it retries against, making throttling look like an outage. The diagnostic: send a full-history and an empty-history request in the same second — if the empty one fails too, it's throughput, not state.
- Block-by-block token accounting showed the identity/style block was 65.1% of the prompt (median of 30 runs), while user-describing content was just 0.1%. Most of the ceiling went to telling the model who it was.
Open questions posed: when a background queue saturates a token ceiling, drop tasks or queue them? Dropping keeps latency honest but silently loses work; queueing preserves work but turns the limit into an unbounded hang. Also: what prompt cuts survived quality contact, and does routing cheap turns to a smaller model actually help or just move the ceiling?
More from coding & agent
- The hard part of building a background agent is building the background, not the agent — AAAzzam · 2026-09-05
- Grok Bot launches template marketplace; in-house procurement agent saved $100K in a week — aryamankhawow · 2026-09-05
- Lindy Launches CC Scheduling: Just CC the AI Agent on Any Email Thread to Book Meetings — HeyToha · 2026-09-05
- Claude Code criticized: no full conversation history or search in VSCode — Birchlabs · 2026-09-05
- Tencent's 770B Hy4 preview builds a playable 3D game from one prompt in agent test — HeyToha · 2026-09-05
- Raspberry AI turns plain-English requests into fashion renders and tech packs on LangGraph — LangChain · 2026-09-05