DeLoop: Repeated Actions as Self-Supervision for Long-Horizon Agents
Tianle Chen, Pushkar Singh, Ketan Todi, Biswajit Paria, Qijia He, Wenqi Wang, Yuwen Tan, Jike Zhong, Sugato Basu, Deepti Ghadiyaram, Arjun Akula
cs.AI
2026-10-14
DeLoop runs DPO on repeated calls versus later recoveries, with no reward. On compressed CodeGym, Qwen3-8B loops fall from 57.3% to 6.1%, steps about one third, success flat.
Long-horizon agents re-issue a tool call they already made and receive the same observation. DeLoop calls that step a loop. Canonicalize the call so whitespace and argument order do not matter, attach the observation, and if that signature appeared earlier, the step is a looping action. A repeat that returns a new observation is not a loop. The loop rate is the share of runs with at least one such step.
Full history does not remove it. On CodeGym, Qwen3-8B loops in 29.0% of runs with the raw history, 57.3% when history is summarized at 4,096 tokens, and 64.7% at a 2,048-token threshold. Gemini 3.5 Flash-Lite moves from 34.0% with no compression to 47.1% under summarization.
Loops and failure show up together. The link is not yet causal. Across six compressed Qwen3 and Gemini settings, looping runs fail 59.5% of the time and loop-free runs fail 33.4%. Qwen3-8B is 85% against 68%. Gemini 3.1 Pro is 48% against 12%. Looping Qwen3-8B runs average 6.5 wasted calls, 10.5 more steps than loop-free runs, and about 11k extra transcript tokens.
A record sitting in context is often ignored. Delete the most recent completed call and Qwen3-8B re-issues it in 48.8% of samples, against 3.3% when the record stays. With the full history, 29.0% of runs still loop. Under compression, 25.2% repeat a call that is still in the context word for word. Summaries that keep an action ledger or an action trace still loop in 52.0% to 62.7% of runs, next to 57.3% for ordinary summarization.
The label is mined from the rollout. No task reward, gold answer, or LLM judge.
On a compressed training trajectory, the looping action is the rejected response. The chosen response is the first later action that is not a loop and is not the same call. Both are paired with the history from before the loop. If the agent never breaks the loop, that step yields no pair. One rollout can yield several. Of 8,000 CodeGym pairs, 82.6% come from failed runs. In a grid task that counts clusters of online servers, the model calls FindUnvisited(3,0) three times, gets (3,0) each time, and only later calls MarkVisited(3,0). The repeated search is rejected. The later mark is preferred.
The preferred action is whatever the model itself did next. SFT on that action alone leaves the loop rate at 37.7%. Contrasting it with the looping action brings the rate to 6.1%. Online GRPO with a per-step penalty on loops reaches 15.0%.
Training is DPO against a frozen reference policy, with β = 0.05. Qwen3-8B trains one LoRA epoch (rank 32) on 8,000 of 9,403 self-mined pairs. The 14B and 32B models train two epochs on 2,331 and 2,244 pairs from their own rollouts. At inference every action comes from the trained policy. The same weights write the summary, with no separate summary loss. Once the history estimate exceeds 4,096 tokens, the harness keeps the system prompt, the task, and the six most recent messages, and replaces the middle with a summary.
DeLoop-Mix adds 4,635 pairs mined with the same rule from ALFWorld, Open-AgentRL, and Terminal-Bench.
Qwen3-8B is evaluated on 300 held-out CodeGym tasks, summarizing once history passes 4,096 tokens. Full history, the compressed baseline, DeLoop, and DeLoop-Mix are averages of three runs.
| Method | Tasks with a loop | Success | Steps/task | Tokens/task |
| Full history | 29.0% | 31.7% | 12.23 | 24.3k |
| Compressed baseline | 57.3% | 25.1% | 15.00 | 23.9k |
| Retrieval memory | 42.7% | 27.7% | 13.34 | 26.1k |
| ACON | 46.3% | 29.3% | 14.84 | 20.5k |
| Duplicate blocker | 46.7% | 25.3% | 14.71 | 24.9k |
| Gold-label DPO | 22.3% | 25.7% | 8.10 | 11.3k |
| Full-history distillation | 43.7% | 26.3% | 11.52 | 26.6k |
| DeLoop | 6.1% | 25.9% | 4.67 | 12.7k |
| DeLoop-Mix | 10.1% | 30.3% | 5.94 | 9.6k |
The loop rate falls from 57.3% to 6.1%, about one ninth of the baseline, a 9.4× cut. Steps fall from 15.00 to 4.67, and tokens per task fall by 47%. Success moves from 25.1% to 25.9%, still short of 31.7% with the full history. Gold-label DPO, which builds pairs from task outcomes, still loops on 22.3% of tasks, takes 8.10 steps, and reaches 25.7% success. The blocker's 46.7% counts proposed repeats, including calls the harness refuses to run.
| Model | Loop rate | Success | Steps/task |
| Qwen3-14B baseline | 42.9% | 45.1% | 15.28 |
| Qwen3-14B DeLoop | 12.3% | 52.3% | 8.18 |
| Qwen3-32B baseline | 38.5% | 37.7% | 13.54 |
| Qwen3-32B DeLoop | 12.9% | 44.8% | 8.82 |
Success rises by 7.2 points at 14B and 7.1 points at 32B. Compression had cost those two sizes about 8.9 and 9.0 points, and DeLoop wins most of that back, with 35% and 29% fewer tokens per task. Qwen3.5-9B goes from 54.7% loops and 49.7% success to 26.0% and 55.3%. Llama-3.1-8B goes from 43.7% loops and 17.51 steps to 2.0% and 5.72 steps, while success only moves from 17.8% to 19.4%.
Among repeated calls, the share that returns a new observation rises from 29.9% to 69.8%. Looping actions fall from 24.9% of steps to 2.5%. Informative repeats fall from 10.6% to 5.8%.
DeLoop-Mix lifts compressed success from 25.1% to 30.3% and full-history success from 31.7% to 36.2%, with a 10.1% loop rate in both settings and 9.6k tokens per task, the lowest row in the table. Cross-domain pairs alone leave the loop rate at 55.3%. Across five context budgets, DeLoop stays between 4.7% and 12.8% loops, while the untrained policy reaches 65.3%. Under seven memory policies, DeLoop-Mix cuts loops from the 42.7%–62.7% band to 6.3%–20.7%. With an action ledger the rate falls from 62.7% to 11.3% and success rises by 5.7 points.
Loops fall in four further environments. ALFWorld and Open-AgentRL are scored on games that were not used to mine pairs: ALFWorld loops go from 55.2% to 27.6% and success from 37.3% to 38.1%; Open-AgentRL from 79.0% to 73.5% and success from 43.9% to 45.5%. AppWorld loops fall from 73.7% to 40.9%, but success falls from 7.0% to 5.8%. ScienceWorld falls from 96.5% to 83.5%, with success from 3.5% to 1.5%. On Terminal-Bench, unproductive duplicate calls drop from 16.6 to 6.9 per task. On all 830 BrowseComp-Plus queries, DeLoop-Mix cuts the loop rate from 22.4% to 1.2% and matches untrained accuracy; CodeGym-only DeLoop reaches 6.4%. Qwen3-32B trained only on CodeGym transfers zero-shot to ALFWorld: loops 64.9% to 54.5%, success 33.6% to 42.5%.
Off the agent harness, scores barely move. LiveCodeBench, 182 problems, greedy pass@1: 53, 52, and 54 solved. IFEval strict accuracy: 82.8%, 83.5%, 83.4%. MT-Bench: 4.225, 4.463, 4.213.
The usual fix is a different memory: an external store, a rolling summary, or a learned compressor. DeLoop leaves the compressor fixed and changes whether the policy uses the record it already has. The only label is an exact match between a call, its observation, and an earlier step.
At 8B the saving is interaction cost. Steps drop to about one third. Success stays next to the compressed baseline and does not recover the full-history 31.7%. At 14B and 32B, most of the success lost to compression comes back.
Refusing a duplicate at execution time does not stop the proposal. The blocker sits at 46.7% loops and 14.71 steps, against 15.00 for the baseline. The share of tasks that still propose an already executed call falls from 71.3% to 13.6% with DeLoop, against 69.8% with no blocker.
A pair exists only when the trajectory later escapes. An unbroken loop is discarded. Eight thousand pairs and one epoch take the 8B loop rate to 6.1%. At 9.4k pairs it levels off at 6.7%.
A drop to about one ninth of the 8B loop rate comes with a 0.8-point gain in success. If loops were the main cause of failure, those two numbers would move together. Clearing the spin saves steps. Solve rate moves at 14B and 32B. Nothing here breaks the loop while holding the rest of the behavior fixed.
The preferred action is only the first later non-looping call that differs, and 82.6% of pairs come from failed runs. On 250 held-out loop prefixes, greedy re-issue of the looping action falls from 15.6% to 6.0%, copies of the recorded recovery stay between 18.8% and 20.8%, and Done() rises from 3.6% to 12.4%–26.0%. At learning rate 3×10⁻⁵ the loop rate is 0.7%, steps are 1.60, and success is 19.3%. At β = 0.01, steps are 2.06 and success is 16.3%. The default recipe still holds success. Push it and the agent simply does less.
On AppWorld and ScienceWorld, loops fall and success falls with them, from baselines of only 7.0% and 3.5%. Open-AgentRL loses just 5.5 points of loop rate.
The detector demands an exact string match. A wasted call written differently is neither counted nor used as a negative. OpenHands flags a stuck agent after four identical returns; DeLoop counts one. Summary writing shares the trained weights, and summary quality is not measured on its own. The 14B DeLoop number is a single run. General ability is checked on three non-agent benchmarks only.