LongHarness Bench: Stress-Testing Language Model Harnesses for Long-Context Reasoning
Quang Hieu Pham, Thuy Duong Nguyen, Jocelyn Qiaochu Chen, Xi Ye
cs.CL
2026-09-30
LongHarness (Alberta) scores four strategy-heavy long-context tasks. GPT-5.6-sol + mini-swe-agent tops out at 68%; on memos RLM costs 12.4× mini-swe-agent and is less accurate.
LM harnesses wrap a base model with search, context slicing, tools, and extra model calls so it can work over long inputs. Current long-context scores barely separate those wrappers. On a 90K-128K LongBench-v2 slice, five GPT-5.6-sol setups sit between 60% and 68%. On OOLONG-Synth every agentic harness beats direct reading, 70%-86% versus 64%. Accuracy saturates, and cost looks similar enough that strategy does not show up. Tasks that yield to one-shot retrieval or independent segment processing never ask whether a harness can change its search plan mid-reasoning.
LongHarness, from the University of Alberta and Amii, scores both accuracy and dollars. The four tasks mix lexical search with semantic matching, and what you should retrieve next depends on what you just found. Most of the context is topically related; only a thin slice helps at each step. Exhaustive scan and selective search can both be correct. They do not cost the same.
Contexts run about 80K-128K tokens. Each instance gets a 3M-token budget across all calls. Scoring is exact: the full ID set or the exact final value. Gold answers are computed on a private structure, then rendered into shuffled public text with controlled distractors, so exhaustive and selective strategies land at different costs.
The comparison set is direct long-context inference plus RLM, OpenCode, mini-swe-agent, and ReAct (BAAI/bge-m3 embeddings). Models: GPT-5.6-sol, Gemini 3.8 Flash, GLM-5.3, Qwen3.8-27B, Kimi-K2.6. The first three use high reasoning effort; Qwen3.8-27B uses medium; Kimi-K2.6 runs in reasoning-enabled mode.
The best combo is GPT-5.6-sol with mini-swe-agent: 68% macro accuracy at $0.816 per instance. Direct inference on the same model is already 60.0% at $0.822, an 8-point lift. No harness wins all four tasks. Best per-task accuracy ranges from 100% on memos to 52% on program pairs.
| Setup | Macro acc. | Cost / instance |
| GPT-5.6-sol + mini-swe-agent | 68.0% | $0.816 |
| GPT-5.6-sol direct | 60.0% | $0.822 |
| GPT-5.6-sol + RLM | 53.5% | $4.30 |
| GLM-5.3 + mini-swe-agent | 42.5% | $3.00 |
| Qwen3.8-27B + OpenCode | 38.5% | $7.06 |
| Kimi-K2.6 best | 1.5% | $5.65 |
Matched accuracy can hide an order-of-magnitude cost gap. On memos, GPT-5.6-sol mini-swe-agent hits 100% at $0.583 by dumping facts into a table and checking claims with a local script. RLM gets 92% at $7.23, 12.4 times more. On execution tracing, direct inference is already 94% at $0.648 with 0.121M input tokens; OpenCode reaches 96% with 1.29M input tokens and $1.37.
Harnesses also hurt strong models. For GPT-5.6-sol, OpenCode drops memos from 44% to 10% and program pairs from 44% to 0%. RLM scores 2% on pairs at $7.86, versus 44% at $0.780 direct. Weaker models gain more: Qwen3.8-27B goes from 0% direct to 38.5% with OpenCode, and from 0% to 92% on tracing at about 23.66 times the cost; GLM-5.3 goes from 9.5% to 42.5%. Kimi-K2.6 never exceeds 1.5%.
The older suites flatten this picture. Same GPT-5.6-sol: LongBench-v2 stays in a 60%-68% band; OOLONG-Synth rewards every harness, with mini-swe-agent at 86% for $0.217. On LongHarness, every harness helps GPT on some tasks and hurts it on others.
You are scoring a model-harness pair, not a model. Extra inference is neither necessary nor sufficient. mini-swe-agent wins memos because it builds a reusable relation table and skips rereading the packet. ReAct's embedding search helps only on program pairs (44% to 52%), where a retrieved item is itself a candidate. Memos need a three-memo evidence chain; similarity search scores 6%.
If you run agents on about 128K documents, do not assume a coding harness beats stuffing the context. For GPT-5.6-sol, direct inference already hits 58% on constraints and 94% on tracing. The wrong wrapper costs more and scores worse.
This is a useful evaluation increment: synthetic tasks, deterministic keys, cost plotted next to accuracy. It does not introduce a new harness.
There is no Limitations section. Several caveats sit in the methods.
Gold pairs mean both programs pass the original tests, not formal equivalence on every input. Tracing is behavioral checks on explicit input contracts. The worlds are tidy: exactly five people, fifteen conflicting memos, five program pairs. Real packets are messier. The LongBench-v2 comparison is a 50-example, hard-heavy 90K-128K slice that the paper says does not match the full benchmark's mix. Costs use public API rates; self-hosted Qwen is a rate-equivalent. Kimi-K2.6 collapses across the board with almost no diagnosis of capability, harness fit, or budget. Main text never states the item count per task; the percentage steps in Table 2 look like a 50-example grid.