DISCO: Distributed Long Context Scaling with Grounding-Reasoning Disaggregation
Guanzheng Chen, Viet Dac Lai, Subhojyoti Mukherjee, Branislav Kveton, Seunghyun Yoon, Franck Dernoncourt, Qizhe Xie, Trung Bui
cs.CL
2026-09-27
DISCO parallelizes grounding on Worker LLMs and plans with a GRPO Driver. Qwen3-8B holds 78.4% on 1M RULER-QA; LongBench v2 Long is 9.8 points above full context.
Million-token context windows are now a spec-sheet number. Effective reasoning length is not. Quality drops as the prompt grows; the literature calls this context rot. RULER-style tests keep showing the same pattern: the nominal window is still there, scattered facts go missing, and multi-hop reasoning fails first.
The workload splits. Contextual grounding, finding the relevant facts, grows with the haystack. Reasoning over those facts usually does not. A single Transformer runs both jobs in one attention window, so the search load burns the bandwidth that multi-hop reasoning needs.
Workarounds fail on one side or the other. RAG and agentic memory hand grounding to embeddings or BM25 and miss implicit evidence. Chain-of-Agents scans chunks with a full LLM, in series, so latency grows linearly and state decays across hops. LLM×MapReduce parallelizes the scan, then reduces once, while each worker still faces the full query. DISCO wants deep, parallel grounding that is structurally cut off from the reasoner.
DISCO treats long context as distributed data, on the Apache Spark analogy: an LLM's reliable window is a node's RAM. When data overflows, you add shards. You do not build a bigger node.
The context is split into K disjoint shards of 4096 tokens, each held by a Worker. The default Worker is Qwen3-4B-Instruct-2507. It only does local grounding. The Driver is a stronger reasoner (Qwen3-8B/14B or Gemini-3-Pro-Preview in the experiments) and never reads the raw long document. It only sees evidence the Workers return.
The Driver compiles the user query into a DAG (directed acyclic graph) with two node types:
The right graph is not known up front. The Driver loops through plan, execute, evaluate, at most 10 rounds, and redraws the DAG around remaining gaps. Longer context adds Map nodes. Harder reasoning adds Wide nodes.
There is no gold DAG, so the Driver is trained with GRPO. A trajectory is several planning steps plus a final answer. The reward has three parts: answer correctness Racc in {0,1}; evidence support Revd in {0,1}, judged by Qwen3-235B-A22B, which penalizes answers that are right for the wrong receipts; format Rfmt in {-1,0} for illegal DAGs. Terminal rewards are broadcast to every planning step. Training data is 4.5k MuSiQue items expanded by recursively inlining Wikipedia pages, contexts 16K-512K, two epochs, learning rate 1e-6.
Baselines: full-context read, RAG with Qwen3-Embedding-4B (top-16 chunks of 256 tokens), Chain-of-Agents, LLM×MapReduce, Recursive LM. Open-weight models get YaRN to 128K.
On RULER-QA, DISCO with Qwen3-8B still scores 78.4% at 1M tokens. RAG in the same setting falls to 10.9%. Full-context cells are blank from 256K up; those runs do not fit. The strongest scalable baseline is LLM×MapReduce, 72.4% at 1M on 8B (75.3 / 73.2 / 72.4 at 256K / 512K / 1M). DISCO is 6 points higher. Accuracy stays roughly flat from 256K to 1M. That is the main evidence that rot is contained.
On LongBench v2 Long (>128K), the gap looks like planning, not needle search:
| Method | Qwen3-8B Long | Qwen3-14B Long |
| Full Long Context | 32.4 | 38.9 |
| LLM×MapReduce | not listed | 33.7 |
| DISCO | 36.1 | 48.7 |
| DISCO (RL) | 39.8 | 50.9 |
14B DISCO is 9.8 points above full context and 15.0 above MapReduce. On Medium (32K-128K), 14B DISCO scores 41.8 against full context 42.3. The win sits in the long bin. 8B full context is 28.8 / 32.4 / 30.6 on Medium / Long / Avg; DISCO is 34.4 / 36.1.
Thinking Mode adds only 0.9 (8B) and 5.6 (14B) points under full context. Under DISCO the same switch adds 5.5 and 13.5. Workers already compressed the evidence, so the thinking budget goes to reasoning.
Scaling Workers from 4B to 14B moves accuracy from 50.9% to 53.8%, cuts average stages from 3.96 to 3.02, and leaves latency about even (415s vs 432s). Scaling the Driver from 8B to 14B moves Long accuracy from 39.8% to 50.9%. The model that should grow is the planner.
With Gemini-3-Pro as Driver on LongBench v2 Long, DISCO scores 71.3% against 72.2% full context, while evaluation cost falls from $126.7 to $20.4, about 6.2×. Under an 8×H100 iso-compute budget (4 GPUs on a 14B Driver, 4 on 4B Workers), latency stays close to flat out to 1M; CoA and full-sequence scans climb linearly.
Reward ablation on 8B, LongBench v2: full reward 39.8% at 381s; no evidence reward 36.3% at 432s; no format reward 34.4% at 499s, because illegal DAGs retry.
This is an inference-time system. You can wrap an existing model. It fits jobs where the document outruns the effective window, the question is multi-hop, and multi-model latency is acceptable: long-report QA, multi-file search, book-length questions. 4B reads, 14B thinks. The expensive model does not run attention over a million tokens.
If you already scan documents in a MapReduce pattern, the gain is incremental. Needle tasks on RULER move by 6 points. The larger lift is on LongBench-style planning, and on Thinking Mode finally having a clean workspace. RL on the Driver is worth doing. The training distribution is synthetic Wikipedia inflation; do not assume the same numbers on logs or codebases.
End-to-end latency of 381-432 seconds is batch-friendly, not interactive.
There is no dedicated Limitations section. The conclusion says DISCO effectively eliminates context rot. That overreaches. The flat RULER-QA curve is real. On LongBench Medium, 14B DISCO still sits slightly below full context (41.8 vs 42.3). Rot is contained in the ultra-long bin, not at every length.
The Driver never reads the source. If the query needs a global co-occurrence statistic or a cross-shard pattern that was not decomposed into entity lookups, Workers will not surface it and the Driver cannot recover. Shard size is fixed at 4096 tokens; evidence that straddles boundaries is not tested.
Training uses 4.5k MuSiQue items with recursive Wikipedia inlining. That looks like entity needles buried in wiki text, which is a different noise profile from contracts, code, or chat logs. RL gains show up mainly on LongBench; the 78.4% at 1M RULER-QA is the non-RL DISCO run.
Matching Gemini-3-Pro-Preview means wrapping Gemini as the Driver: 71.3% vs 72.2% full-context Gemini, not Qwen3-14B catching Gemini. The cost comparison is only reported in that API evaluation. Recursive LM collapses to single digits at 8B/14B. The paper already notes that RLM wants models with hundreds of billions of parameters and coding skill. That baseline is weak.