Heartbeat Thinking: Speculative Reasoning and Tool Execution While Listening in Realtime Voice Agents
Zhenyu Zhang, Kewei Fu, Yongqi Tong, Chenzhi Zheng, Wei Wu, Jiang-Ming Yang, Jianshe Li
cs.CL
2026-10-19
Heartbeat finishes tool calls while the user still speaks. On FDBench v3, coverage rises from 34.67% to 50.67% and response latency falls from 6.90s to 4.39s.
Tool-using voice agents usually wait until the user stops talking. An order id, a budget, or the item to look up is often already in the first half of the sentence. Cascaded pipelines still wait for VAD (voice-activity detection, the check that the turn is over), then run reasoning, tools, and speech synthesis only after the utterance is complete. The pause the user feels sits after the speech ends.
SHANKS moves reasoning forward, but each generation is cut on a speech-chunk boundary, so tool arguments may never be finished. SIA interrupts in-flight generation when a new event arrives, and a call that has not been issued is wasted. The more often the transcript updates, the less uninterrupted time the model has to finish one call. Early work also has to survive the rest of the request. A budget added later can invalidate an earlier search.
Heartbeat Thinking, from Ant International, does not cancel computation that is already running. New speech goes to the next Thinker invocation. After the user stops, a separate Responder answers the full request from preparation that has already finished. The two roles can be different models. The main runs use Qwen3.6-35B-A3B as Thinker and DeepSeek-V4-Flash as Responder.
The timer is fixed. A new Thinker is attempted every 500 ms, with at most two running. If both slots are full, that tick is skipped, not queued. Each invocation reads the partial transcript, completed reasoning, and returned tool results as of its start. One Thinker can finish arguments while another absorbs the latest fragment.
The Thinker emits one of three outputs. noop means nothing new to do. It can also record reasoning only, or attach a structured tool request. Reasoning is written to a shared record before tools run asynchronously, so model time and tool time overlap. The prompt forbids repeating a pending or completed call with the same tool name and full arguments. Later invocations use returned results to fill missing arguments and fire dependent calls. Preparation can advance even when the transcript does not change.
When VAD marks the end of speech, new Thinkers stop. Unfinished speculative work, including queued tools, is cancelled. The Responder receives only the snapshot completed by time c. Later returns are dropped. In a ReAct loop (reason, then act) it drops interpretations that later speech contradicts, reuses results whose name and full arguments match, and reissues calls that still have no result. It does not wait on cancelled jobs.
Window Thinking keeps the same timer and drops outputs that arrive after their window, as a stand-in for SHANKS-style truncation. Speech-to-Speech, Pipecat, and LiveKit wait until speech ends, so their speculative coverage is 0 by definition.
SpeechFCEval has 1,914 samples scored on function names and arguments. FDBench v3 has 100 recordings, 79 scenarios, and 12 speakers, and tests intent under disfluency and revision. AudioCRAG has 1,312 knowledge questions. STC, speculative tool coverage, is the share of reference calls that finished during listening, entered the Responder, and were not repeated with the same tool name and full arguments. On AudioCRAG it is the share of samples that reuse a finished speculative retrieval and do not retrieve again.
| Method | SpeechFCEval Acc / STC | FDBench Quality / Pass@1 / STC | AudioCRAG Correct / STC |
| Speech-to-Speech | 58.66 / 0 | 84.0 / 75.0 / 0 | 56.48 / 0 |
| Pipecat | 56.83 / 0 | 78.0 / 63.0 / 0 | 56.94 / 0 |
| LiveKit | 52.85 / 0 | 77.6 / 67.0 / 0 | 59.45 / 0 |
| Window Thinking | 67.95 / 33.13 | 72.0 / 57.0 / 34.67 | 57.24 / 20.05 |
| Heartbeat | 66.85 / 43.69 | 77.0 / 65.0 / 50.67 | 55.49 / 62.42 |
Against Window Thinking, STC is higher by 10.56, 16.00, and 42.37 percentage points. On FDBench, quality moves from 72.0% to 77.0% and Pass@1 from 57.0% to 65.0%. That Pass@1 is still below Speech-to-Speech at 75.0% and LiveKit at 67.0%, and above Pipecat at 63.0%. Quality at 77.0% sits under all three cascaded baselines (84.0%, 78.0%, 77.6%). SpeechFCEval accuracy falls from 67.95% to 66.85% and remains above the cascaded systems. On AudioCRAG the correct-answer rate falls from 57.24% to 55.49%, and the missing-answer rate rises from 14.33% to 16.84%. Coverage went up. Task scores did not follow.
FDBench latency is measured from the speech endpoint. Negative values are before the user finishes. Both models had thinking disabled for this measurement.
| Method | Correct tool-call start | Final-response onset |
| Heartbeat | -4.37 s | 4.39 s |
| Window Thinking | -2.74 s | 6.90 s |
| Pipecat | 2.45 s | 8.08 s |
| Speech-to-Speech | 2.72 s | 8.39 s |
| LiveKit | 3.52 s | 8.52 s |
Final-response latency falls from 6.90 s to 4.39 s, down 36.4%, and 45.7% below Pipecat, the fastest cascaded baseline at 8.08 s. The first tool call matching the request starts 4.37 s before speech ends under Heartbeat, 2.74 s before under Window Thinking, and 2.45 to 3.52 s after under the cascaded baselines.
The STC lead survives a Responder swap, and quality can reverse. On SpeechFCEval, Qwen3.8-Flash-Next reaches 64.30% accuracy and 55.27% STC, above Window Thinking at 57.14% and 44.05%. Qwen3.5-397B-A17B reaches 69.70% accuracy versus 67.27%, but only 41.20% STC versus 32.49%. The Responder with higher coverage has lower accuracy. On FDBench that larger model scores 61.0% Pass@1 with Heartbeat, below Window Thinking at 62.0%, with STC at 45.33% versus 34.67%.
Replacing the Thinker with Qwen3.5-9B, GLM-4.7-Flash, or Gemma-4-26B-A4B-it still produces nonzero STC. With DeepSeek-V4-Flash answering, Gemma's SpeechFCEval accuracy is 68.58%, above the default Thinker's 66.85%, while STC is 42.71% versus 43.69%. On AudioCRAG, Gemma's reuse rate is 74.70% and its correct rate is 55.87%, still under Window Thinking's 57.24%.
On FDBench, after leading and trailing silence are removed, a call consistent with the final request starts before speech ends in 73.3% of recordings whose target falls in the first 70% of the utterance, and in 49.2% when it falls in the last 30%. After an explicit revision, 70.6% issue a corrected call before speech ends. Another 4.0% have correct arguments and a stale answer. A $50 budget is still unspoken when a $99.99 coffee maker is added to the cart. A later search finds a $40 item. The answer says the cheaper product was added and still reports $99.99. The second search does not undo the cart update.
Coverage rises in four of five SpeechFCEval groups. Speech-BFCL parallel calling goes from 36.34% to 58.03%. SmartInteract falls from 40.57% to 38.32%.
Seconds of user speech can become tool results that are already back. Against chunk-bounded generation, Heartbeat adds time to finish arguments, and tool jobs that the next speech update does not kill. Thinker and Responder hand off through an external record and can be different models. The reported setup already splits them: Qwen3.6-35B-A3B while listening, DeepSeek-V4-Flash to close the turn.
The gain is in scheduling. On FDBench the final response is 4.39 s versus 6.90 s for Window Thinking, and Pass@1 is higher there too. Non-speculative Speech-to-Speech still leads Pass@1 at 75.0%, LiveKit is at 67.0%, and Heartbeat is at 65.0%. Early calls pull the spoken answer forward. Correctness depends on whether the Responder reads the snapshot correctly.
Cost is unreported. At 500 ms and two concurrent Thinkers, a long utterance triggers repeated inference, with no token or GPU-time comparison. A cart update or an order placed during speculation cannot be undone by cancelling unfinished jobs at the handoff.
The paper lists three limits. The interval and the concurrency cap are fixed, with no adaptation to transcript changes or tool completion, so quiet stretches still spend extra inference. The Responder sees only the handoff snapshot, and later tool results miss the turn. Latency is measured only on FDBench v3, so the trade among coverage, inference cost, and response time under other tool workloads is open.
Higher coverage is not better task quality. SpeechFCEval and AudioCRAG both decline versus Window Thinking. In 4.0% of FDBench recordings the answer uses a stale result despite correct arguments, and a cart write cannot be rolled back. There is no follow-up with a changed prompt or handoff.
Window Thinking drops late outputs on Heartbeat's timer. It approximates SHANKS truncation. It is not a full rerun of that system, and SIA and Stream RAG have no head-to-head numbers here. FDBench has 100 recordings, so the Pass@1 move from 57% to 65% is thin. The judge is DeepSeek-V4-Flash at maximum reasoning effort, the same family as the main Responder, and any bias toward this handoff is unchecked. STC treats a repeated call as uncovered, so a Responder that reissues the same call lowers the score. Across models, that mixes answering habits into the system result.