Ready Cohorts: Bounding GPU Opportunity and Avoiding Host Round Trips in LLM-Agent Control
Josef Liyanjun Chen
cs.DC, cs.AI, cs.OS
2026-08-12
On 851 real agent sessions only 43% of control steps pack onto the GPU under ideal sliding-deadline batching, collapsing to zero below 10k sessions; keeping a route decision on-device beats a host round trip by 1.19x to 2.39x, but only when a matched host decision is removed.
Between every model call and tool call, an LLM agent runs a small deterministic control step: parse the previous outcome, advance a state machine, check budget and policy, pick the next route, emit the next action. Each step is cheap next to inference, but a service running thousands of concurrent sessions crosses the CPU-GPU boundary on every one of them. Production characterizations have found this host-side orchestration sitting on the critical path and creating bursty CPU demand. The natural question is whether, since inference is already on the GPU, those control steps could be batched and run on the device too. This paper refuses a slogan-level yes or no and splits the intuition into two separately measurable, falsifiable gates: supply (can the same kind of work accumulate past the GPU's batch threshold before its deadline) and observation placement (how much of the control chain proceeds without exposing an intermediate decision to the host).
The unit of study is the deterministic control transition, smaller than an LLM request, a tool call, or a workflow. It reads typed state and a ready event and emits new state, a route, and optionally an effect descriptor. Model inference, tool execution, and the privileged commit of external effects stay out of scope: a GPU may compute that a tool should be called, but the paper does not let it hold cloud credentials, spawn VMs, or commit side effects.
The first study defines four quantities that must not be substituted for one another:
They satisfy F no greater than P no greater than U, and A no greater than P. The trace is 851 real customer-service sessions (airline, retail, telecom) from the Exgentic tau2 dataset, 9,031 LLM spans, with each span completion treated as one candidate control event and the route key derived from the recorded outcome (final text, error, named tool, multi-tool), giving 70 route labels. Sessions are replayed with stationary Poisson arrivals to a target concurrency C, and the primary cell is frozen at C=100,000, K=256, and a 50 ms deadline before any P is computed.
The second study is a mechanism experiment that isolates the second gate: once a decision is already resident on the GPU, should it return to the host? Three mechanisms run on one frozen CUDA program (16-byte synthetic state per agent, two route bodies, one global binary decision per epoch). Host round trip computes the predicate, copies the four-byte result back, synchronizes, and launches the chosen route graph from the host. Device resident launches one root graph; a one-thread selector reads the predicate on device and tail-launches the matching path graph, running for H epochs without ever exposing the decision to the host. A no-decision floor replays the oracle sequence with the predicate and selection omitted; it is a structural floor, not a deployable policy.
The primary trace cell gives F=30.19%, P=43.00%, U=45.85%. Exact packing recovers 81.83% of the opportunity that fixed windows leave on the table, with P sitting 2.85 points below U. What matters more is how the boundary collapses: at K=256, P is zero for every deadline when C is at most 10,000, and even at C=100,000 it is still zero under 10 ms and 25 ms deadlines. A large nominal swarm does not mean a route sees a profitable cohort inside a short budget.
Lowering K matters more than admitting boundary-crossing cohorts. At C=10,000 and 50 ms, P is 22.2% for K=32 but 0.0% for K=64; at C=100,000, K from 32 to 256 gives 66.8%, 66.0%, 48.4%, 43.0%.
In the mechanism study, device-resident beats host round trip in all 36 placement-cells, with ratios from 1.19x to 2.39x. At the primary cell (N=256, H=32):
| Placement | GPU | Resident / Host |
| Local | GTX 1660 Ti | 1.71x |
| Modal | L4 | 2.39x |
| RunPod | L4 | 2.06x |
| Lambda | H100 SXM5 | 1.84x |
The absolute saving is 194 to 363 microseconds per 32-epoch cohort. The resident path is still 6.60x to 8.17x slower than the oracle floor, an undecomposed gap from predicate, selector, and graph overhead. All 14,557,440 tested invocations are field-exact and decision-exact against an independent host oracle.
The decisive control is a negative one: moving the launch from the host into a GPU kernel while removing no host decision is slower in all 60 cells across five placements (1.07x to 1.99x). Launching from the device buys nothing on its own; the speedup comes only from removing one matched host-mediated decision epoch.
The paper turns the question of whether agent control can run on the GPU from a slogan into two separately measurable quantities, and labels the inference boundary of each. For engineers building agent serving, the most actionable conclusion is the negative control: do not expect a kernel-issued launch to save anything on its own. Device-resident decisions pay off only when they also eliminate a decision round trip that would otherwise return to the host.
It also pours cold water. The two studies' numbers cannot be multiplied: the trace's K=256 is a swept candidate, not the measured crossover for the resident mechanism, and the trace model lacks the H=32 horizon. The paper delivers an opportunity surface (the offline optimum under idealized assumptions) and a mechanism direction, not any service-level speedup. The actual deployment value, A, requires a finite online runtime that does not yet exist.
The authors are unusually candid. The offline optimum assumes future omniscience, zero service time, unlimited capacity, no batch cap, and deadlines on launch rather than completion, none of which hold in a real system. The trace is one 851-session panel across three customer-service domains under forced stationary Poisson arrivals; three seeds measure Monte Carlo variation conditional on that panel and model, not workload-population confidence, and bursty or correlated arrivals would move the boundary.
More fundamentally, the route key is a conditioning proxy, not verified semantic fusion. It omits the state-machine node, schema, arguments, policy context, and multi-tool identities, and the packing arrays drop the per-session sequence constraint. The 43% figure is thus an upper-bound-style opportunity estimate; executing it still requires per-route admissibility checks. The mechanism side tests one global binary decision over a regular synthetic state array; real route bodies still need per-event compaction, variable bodies, ingress, egress, effect ordering, recovery, and CPU fallback, and the timing excludes graph construction, state reset, result copy, and validation. The four named placements confound GPU, provider, host, image, driver, and region, so they support named-placement effect sizes, not hardware-population inference. The metric is batch-average cohort-horizon wall time, not invocation P99 or end-to-end task latency, and there is no tuned CPU baseline, CPU core-seconds, energy, cost, model throughput, TTFT/TPOT, or task utility.