TAC-CoT: Tool-Empowered Implicit Chain-of-Thought via Interleaving Explicit Tokens
Juseon Do, Sangmin Bae, Jeesu Jung, Hwanjun Song
EMNLP 2026 Main Long
cs.CL
2026-11-01
TAC-CoT has small LLMs reason in 3-token latent segments between tool calls. On agentic math and QA it gains up to 8.8 accuracy points while cutting forward passes by up to 6.8x.
Agentic LLM systems pay for their reasoning twice. In a ReAct-style loop the model writes out every intermediate step as text around each tool call, so longer horizons mean more generated tokens and more places for an early mistake to compound. Implicit CoT was supposed to fix this by swapping verbal rationales for a few continuous tokens, but existing methods (CODI, SIM-CoT and kin) consume one fixed-size block of latent tokens before producing any output. There is no point in that block where the model could emit a tool call or ingest an observation, so these methods cannot act as agents, and every query gets the same latent budget no matter how hard it is.
TAC-CoT (KAIST, EMNLP 2026) connects the two worlds: compact latent reasoning segments, with tool calls, observations, and the final answer kept as plain text.
The design is interleaving. Each natural-language reasoning span in a tool-use trajectory is replaced by a segment of b=3 continuous tokens; tool calls, returned observations, and the answer stay verbatim.
The explicit tokens preserve the executable interface an agent needs; per-segment routing lets the budget scale with difficulty. Each choice patches one of the two failures of prior implicit CoT.
Backbones: Qwen2.5-0.5B, Llama-3.2-1B, Qwen3-4B. Baselines: a fine-tuned Explicit-CoT agent, TokenSkip (prunes explicit reasoning), and three fixed-budget implicit methods (PCCoT, CODI, SIM-CoT). Cost is measured as average forward passes per query (#Fwd), independent of hardware and API latency.
| Setting (Qwen2.5-0.5B) | Metric | Explicit-CoT | TAC-CoT |
| GSM8K, no tools | Acc / #Fwd | 50.9 / 117.3 | 58.8 / 54.6 |
| Math average, no tools | Acc / #Fwd | 41.1 / 113.1 | 45.4 / 53.7 |
| QA average, with tools | EM / #Fwd | 42.4 / 283.3 | 45.8 / 44.2 |
On Llama-3.2-1B the QA average moves from 53.4 to 59.2 EM while #Fwd drops from 76.3 to 35.1; on Qwen3-4B the math average goes from 66.1 to 71.3. Averaged over backbones, math gains 5.8 points and QA 2.4 over the Explicit-CoT agent, holding on out-of-domain sets (GSM-Hard, SVAMP, 2WikiMultihopQA, TriviaQA). Attaching real tools lifts QA by 29.9 points on average over the no-tool variant on Qwen2.5-0.5B while cutting forward passes by 13.3x. The headline claims are up to 8.8 accuracy points and up to 6.8x fewer forward passes; measured wall-clock speedups on one L40S GPU are 2.3x (math) and 2.4x (QA).
Two diagnostics carry the argument. Grouping GSM8K by the number of gold reasoning steps, the no-tool advantage over SIM-CoT grows from +4.6 points on one-step problems to +30.3 on four-step problems: fixed budgets fall apart exactly where reasoning gets deep. And swapping explicit segments for implicit ones one at a time (zero through four; four is the 90th percentile in the training set) lifts math accuracy monotonically from 41.1 to 44.9, with 45.4 for TAC-CoT. Verbose traces themselves manufacture accumulating errors, and this ablation is the strongest evidence for it.
Tool-use quality is measured directly. On QA, TAC-CoT's Answer Coverage@5 roughly matches Explicit-CoT's AC@10, so it reaches the evidence in fewer retrieval steps. On ToolBench under GPT-5.1 judging it posts a 51.8% Solvable Win Rate in the single-tool scenario and 58.6% in the multi-tool scenario (p<0.01). The fixed-budget implicit baselines use fewer forward passes (PCCoT averages 9 per math query) but cannot retrieve evidence and degrade sharply on QA. TAC-CoT's position: Explicit-CoT-level accuracy at one half to one sixth of the cost. The absolute cheapest slot still belongs to the fixed-budget methods, at the price of being unusable on QA.
For anyone paying agent inference bills, this is the first complete recipe for running implicit CoT inside a reasoning-acting loop; until now the two were structurally exclusive. The gains concentrate in multi-step, multi-tool trajectories, which is precisely where agent costs hurt. Across 0.5B to 4B models accuracy goes up rather than down while more than half of the forward passes disappear.
The trade-offs are equally clear. You need trajectory construction plus fine-tuning, so this is not plug-and-play, and the readable reasoning trace is gone. Teams that need interpretability lose their debugging handle in the exchange.
The authors flag two: routing operates at segment granularity (three tokens fixed within a segment) rather than token level, and longer interaction horizons remain untested.
Reading the numbers raises more. The with-tools QA gain on Qwen2.5-0.5B averages 3.4 EM, of which 2WikiMultihopQA contributes +5.7 and TriviaQA only +1.5; the abstract's 8.8-point figure is an 'up to' claim whose exact setting the paper body never pinpoints. The HotpotQA training trajectories are synthesized by DeepSeek-V3; the authors argue that all methods share the same trajectories so teacher bias cancels in comparison, which holds, but the synthetic ceiling remains. Wall-clock was measured on a single L40S and lands at 2.3-2.4x, well below the #Fwd ratios, and the paper itself cautions against converting one into the other. Every experiment sits at 4B parameters or below, so whether the recipe survives on larger or reasoning-specialized models is an open question.