Second Thought: Reasoning in Parallel as LLM Agents Act and Observe
Zhensu Sun, Chengran Yang, Yunbo Lyu, Jieke Shi, David Lo
cs.AI, cs.SE
2026-08-14
ReAct agents stop reasoning while serializing actions and waiting for observations; Second Thought fills that idle window with four parallel reasoning branches harvested on arrival, cutting turns in all nine tested settings, main-thread decoding by up to 43%, and measured latency by 10.9%.
A ReAct agent loops through Thought → Action → Observation, and substantive reasoning happens only in the Thought phase. Once the action is serialized and the agent waits for the environment, reasoning freezes: the Action phase just renders an already-formed plan into a tool call, and the Observation phase decodes nothing at all. The paper calls this recurring, unpredictably long interval the reasoning idle window. The slower the tools (container tests, builds, retrieval), the longer the window and the more time spent purely waiting.
Existing parallel-reasoning schemes (Self-Consistency, Tree-of-Thought, multi-candidate sampling) all expand within the Thought phase, putting extra compute on the critical path. This paper's angle is orthogonal: leave the main thread's reasoning alone and relocate the added reasoning into wait time that is otherwise wasted.
The framework, Second Thought, is training-free and lives entirely at inference. The instant the main Thought phase ends, it forks four auxiliary branches that decode concurrently with the Action/Observation phases; when the observation arrives, every branch is terminated and its completed thoughts are harvested into context for the next turn. The four branches span retrospective vs. prospective direction × current-turn vs. full-history scope, each targeting a recurring failure mode:
Two design choices carry the weight. First, atomic thoughts: every branch streams its output as self-contained units wrapped in XML tags, at most 25 words each, with no cross-references, so interruption at any token invalidates only the unit in flight, so no branch-side coordination protocol is needed for a window of unpredictable length. Second, the fork mechanism: instead of spawning an external critic, the same model continues its own full trajectory, branches share the prompt prefix with the main thread (so the KV cache is reused), and branch decoding disables native thinking to emit thoughts as fast as possible. Harvests are capped at five thoughts per dimension; if nothing completes, the agent degrades cleanly to the baseline ReAct loop.
Three agentic benchmarks (SWE-Bench Pro for repo-level bug fixing, Terminal-Bench 2.1 for terminal operations, τ³-bench for banking dialogue) × three reasoning LLMs (DeepSeek-V4-Flash, Qwen3.6-Plus, MiniMax-M3), native thinking modes on, Benjamini–Hochberg-corrected significance tests:
| Setting | Pass@1 (base → ours) | Main-thread decoding (base → ours) | Turns |
| SWE-Pro + Qwen3.6 | 52.0 → 51.3 (ns) | 36,519 → 20,798 (−43%) | 57.1 → 50.6 |
| SWE-Pro + DeepSeek-V4 | 48.7 → 52.0 | 23,841 → 20,255 (−15%) | 56.2 → 52.8 |
| TB2 + Qwen3.6 | 39.3 → 51.7 (+12.4) | 25,158 → 31,396 (+25%) | 25.5 → 24.0 |
| TB2 + MiniMax-M3 | 49.4 → 59.6 (+10.2) | 36,686 → 36,705 (flat) | 44.6 → 43.1 |
| τ³ + Qwen3.6 | 16.7 → 19.8 | 16,755 → 13,764 (−18%) | 26.1 → 25.9 |
Turn count drops in all nine settings; main-thread decoding drops in six, by up to 43% and roughly 20% on average; Pass@1 is statistically unchanged in seven of nine and significantly positive in two (+12.4, +10.2 points), with the single decrease amounting to less than one instance out of 150. The compute-matched control s1 uses budget forcing to push an equivalent reasoning budget onto the main thread's own thought: Second Thought attains strictly higher Pass@1 with 1.3–3.2× less sequential decoding in all four applicable settings. On SWE-Pro + Qwen3.6, s1 inflates output from 36.5k to 65.6k tokens and loses 3.3 points. A paired replay (50 instances, fixed concurrency, median of 3 runs) confirms wall-clock time drops from 256.9 s to 229.0 s (−10.9%), split into −13.4% main-thread decoding and −6.0% tool execution. In ablations, removing Recall costs the most accuracy, removing Rehearse holds accuracy but adds 10% main-thread output, evidence that pre-computation offloads deliberation the main thread would otherwise repeat per turn, and single-branch variants fall near baseline.
This reframes agent latency from "think less" to "think elsewhere." It is training-free, touches neither model nor harness, and adds only a fork/merge layer over streaming APIs. The slower the tools, the bigger the payoff; banking dialogue, with short idle windows and failures rooted in retrieval and policy adherence rather than planning, gains least. For agent-infrastructure work, the atomic-thought output contract is reusable wherever interruptions have unpredictable duration. The cost side deserves attention: four branches raise per-task API cost by 66.4–181.5%, almost entirely from cached prefix reads rather than output; keeping only the strongest branch (Alternative) cuts the overhead to 16.3–35.5%.
There is no dedicated Limitations section; the authors' own boundaries are scattered through the analysis. The τ³-bench asymmetry is attributed to short idle windows and retrieval/policy failure modes rather than planning errors, plausible, but the dynamic branch-selection experiment remains future work. The default configuration is window-starved: an unbounded variant that stalls the main thread until branches close reaches 56.7% (+4.7 over default, capturing 41% of the attainable gain), and that remainder sits on the critical path. The s1 control is missing on MiniMax-M3 and τ³-bench because budget forcing cannot combine with function calling, so the compute-matched claim rests on four settings. Two more things stand out: the gains depend structurally on idle-window length, and thinking models that already produce many candidates inside Thought may have thinner windows to exploit; the replay study covers 50 instances × one model × one benchmark, and the shared-KV-cache deployment assumption that keeps branch cost low depends entirely on how a third-party API prices cached reads.