LongCapHarness: Capability-Guided Auto-Harnessing for Long-Context Reasoning
Tao Zhang, Yuhang He
cs.CL
2026-11-07
No post-training: GPT-5.6-Sol searches code harnesses that lift Llama3-8B from 15.1 to 79.3 on HELMET+RULER, above NExtLong, for about $541.
Recent long-context work first synthesizes long documents, then post-trains. NExtLong packs hard-negative distractors into long sequences. EntropyLong keeps context that predictive uncertainty marks as useful. PolicyLong refreshes the data as the model being trained changes. The data recipe and the training recipe are tuned apart from each other, and the bill is GPU time: about 960 H100 hours for NExtLong, estimated at $3,696, and a FLOP-scaled $1,859 for PolicyLong.
These tasks are also a poor fit for one forward pass over the whole document. A solver has to recognize the task, find evidence, keep intermediate results, and assemble an answer that stays consistent. Inside a single call those steps stay implicit, and the model has to coordinate them while reading everything.
LongCapHarness, from Microsoft Research Asia, does not synthesize data and does not update weights. The claim is narrow: a model with no long-context pre-training can still score well on long-context benchmarks once it is given a program for the task. The program separates total input length from the context budget of any one call, and each call sees only the slice that step needs.
A capability is a transformation that is not tied to one question-answer pair. The same transformation can serve many items. The executable wrapper is a capability harness: code, prompt templates, and execution rules. The harness is split into capability primitives, each with a fixed input and output.
Reranking, the running example, has two primitives. Relevance judgement reads the query and a passage and emits a score or a preference. Ordering turns those judgements into a ranking. Control logic sets the order of calls and the stop rule. Capability state is the cross-question protocol: how a new question is formatted for the primitives, and which intermediate values must be kept. The question and the scratch results of a single run are wiped every time. They are not part of that state.
GPT-5.6-Sol writes the harness. Llama3-8B is the model used while the harness is being tried. The loop is RSI: run the harness, compare with the development-set reference, and edit until the gap is small enough.
At test time Llama3-8B or Qwen2.5-3B only selects an entry, matching the question against those descriptions, then runs the loaded code and templates. Code primitives update records stored outside the context window. Model primitives receive the template, the relevant question fields, and the passages for that step. Primitives used on HELMET and RULER include keyword matching, lookup, relation following, state tracking, counting, ordering, relevance judgement, rule inference, classification, evidence selection, and evidence integration.
The development set has 2,242 examples and the held-out test set 59,826. Every evaluated task type appears in development. The questions do not overlap.
Table 1's Avg is an equal blend of the HELMET category mean (H-Avg) and RULER, on a 0-100 scale. HELMET is a long-context suite; RULER is dominated by synthetic retrieval, tracking, and counting. At execution, Llama3-8B still has an 8K window and Qwen2.5-3B a 32K window. NExtLong, PolicyLong, and EntropyLong are models continued to 128K. RRSI is a recursive-improvement baseline that does not decompose tasks into capability primitives.
| Setup | Avg | Reference |
| Llama3-8B base | 15.14 | 8K window |
| NExtLong | 70.15 | Llama, training est. $3,696 |
| RRSI | 39.67 | Llama, dev est. $332 |
| LongCapHarness | 79.32 | Llama, dev est. $541 |
| Qwen2.5-3B base | 61.91 | 32K window |
| PolicyLong | 64.10 | Qwen, training est. $1,859 |
| RRSI | 38.46 | Qwen |
| LongCapHarness | 78.73 | Qwen, harness transferred |
Llama at 79.32 is 64.18 above its base, 9.17 above NExtLong, and 39.65 above RRSI. Qwen at 78.73 is 16.82 above its base, 14.63 above PolicyLong, and 40.27 above RRSI. Swapping the execution model drops the average by 0.59. The transfer retunes the interface, tokenizer, evidence budget, and serving setup. Code rules stay the same.
The 64.18 point jump is two blocks added together. RULER moves from 17.80 to 94.62. The HELMET category mean moves from 12.48 to 64.01. Against NExtLong the category mean is only 5.21 points higher (64.01 vs 58.80), while RULER is 13.12 higher. Most of the 9.17 point aggregate lead sits in RULER.
Tasks with rules that code can run supply the clear wins. Recall is 100 for both execution models, 71.81 above Llama RRSI and 17.44 above NExtLong, and it holds from 8K through 128K inputs. Qwen's RULER score is 94.28, 17.61 above PolicyLong's 76.67. Reranking leaves the judgement to the model and the sort to code: Llama scores 39.67, above NExtLong's 31.47; Qwen scores 56.28, above RRSI's 21.15.
Semantic tasks stay close. On ICL, Llama is 0.40 above RRSI and Qwen is 2.63 below it. On RAG both trail RRSI, by 4.93 and 0.55. On LongQA, Llama at 36.84 beats RRSI's 31.65 and still sits under NExtLong's 37.30; Qwen at 26.45 sits under RRSI's 27.55.
LongBench-v2 is reported only for Qwen. Medium (32K-128K) is 34.47, above PolicyLong's 27.10. Long (128K-2M) is 27.85, 3.65 behind PolicyLong's 31.50.
The reranking RSI trace changes the implementation, not the task. BM25 plus numeric sort averages 40.35. LLM scoring then sorting drops Llama to 38.95: short inputs rise, long inputs fall. Pairwise comparison with group merging reaches 56.28 on Qwen, and at 128K that version is 26.75 above the previous one. Llama only reaches 39.67. Each of the four LLM runs uses the same 500 items, 100 per length. After the move to Qwen, reranking rises 16.61 and LongQA falls 10.39.
Development cost is an API estimate for GPT-5.6-Sol. LongCapHarness uses 814.11M input tokens and 3.35M output tokens, about $541, against about $332 for RRSI. $541 is under 15% of the NExtLong training estimate and under 30% of PolicyLong's. At inference, Llama output tokens fall from about 89.55M under RRSI to 1.87M, and the estimated charge is $8.23 versus $8.30, while input tokens rise from 236M to 408M. On Qwen the inference estimate rises from $29.39 to $53.74.
RRSI keeps reaching for truncation and character chunking. LongCapHarness edits the solving steps.
If an 8B or 3B model is already in hand and another long-context training run is hard to justify, the budget here is legible. A strong model writes each harness once. The small model selects it and runs it. Counting, lookup, relation following, and state tracking have explicit rules, so an 8K window can cover 128K inputs.
When the model itself has to read and integrate evidence, the lead shrinks. RAG, LongQA, and the longest slice of LongBench-v2 are that case. The part that can be written as code leaves the model. The part that has to understand text stays with the execution model. The writer is not interchangeable either. Scores drop sharply when Llama3-8B or Qwen2.5-3B is asked to build the harnesses.
The development set includes every task type under test, so the measured reuse is on new inputs of seen types. Harness selection is a match between a short description and the question, and no selection-error rate is reported. The stopping rule has no external threshold. The only full RSI trace is reranking. Rounds and regressions for the other primitives are not reported.
On LongBench-v2 Long, inputs reach about 2M tokens and Qwen still trails PolicyLong by 3.65. Finding and merging evidence there is still a semantic step.
The cost columns are not one ledger. Training figures are H100 rental, and PolicyLong further assumes equal effective throughput per GPU. Development figures are API list prices. Figure 1 labels the Llama and Qwen averages 79.97 and 77.91; Table 1 says 79.32 and 78.73. The gap is not explained. For EntropyLong on Llama, the only published number here is RULER at 87.37, which LongCapHarness beats with 94.62.