Tree-Ising: Repairing Language Models with Bit-Level Ising Edits
Hadi Salloum, Ammar Ali
cs.LG
2026-11-10
Tree-Ising recasts an LLM agent's weight repair as per-row Ising optimization over BF16 bits: 54.6% on MATH-500 vs the agent's 51.7%, at $0.90 and zero LLM tokens instead of $312.
Qwen3.5-0.8B in thinking mode gets stuck in short repetition cycles: on 93% of MATH-500 problems it repeats the same text until the 32k token cap, and accuracy sits at 7.9%. Recent MIT work by Takehi et al. showed the fix can be written directly in weight space: a GPT-6 Astra agent read the checkpoint, ran forward passes, and rewrote 64 columns in two MLP output projections, lifting MATH-500 to 48.1%. The bill: 158 turns, 89.7 million tokens, 50 forward-pass requests, 15.6 hours, $312. And since every decision was made in natural language, nothing guarantees the edit is anywhere near optimal.
This paper turns the same repair into a math problem.
Tree-Ising runs in five steps:
The trick that makes it a QUBO at all: pick the best move per weight in advance, so the only remaining decision is binary, apply it or not.
| Setting | MATH-500@4 | GSM8K@1 | Cut at cap (MATH-500) |
| Base model (reported) | 7.9% | 13.8% | 93% |
| GPT-6 agent edit | 51.7% | 65.6% | 31.6% |
| Tree-Ising (annealing) | 54.6% | 70.5% | 23.9% |
| Tree-Ising (bifurcation) | 55.1% | 70.8% | 23.9% |
Under one harness, Tree-Ising beats the agent edit on all five math benchmarks, with MATH-500 and GSM8K significant after Bonferroni correction. (The agent's originally reported 48.1% reproduces at 51.7% here, in line with an independent 51.0% reproduction.) Cost: 4 forward-pass requests versus 50, 31 minutes versus 15.6 hours, $0.90 versus $312, zero LLM tokens. The edit is smaller too: 13% fewer changed weights, 2.2x fewer BF16 bit flips, and by construction it never writes a shared exponent or a mantissa, while the agent edit rewrote about 130k mantissas.
On Qwen3.5-2B, with layers and dose chosen from the base model alone, the same recipe lifts greedy GSM8K from 21.6% to 69.4%, truncation falling from 77.9% to 26.7%, the whole pipeline running in 26 minutes on one A100. When that model finishes, it is right 97.9% of the time; the failure was never capability, it was the inability to stop.
Ablations separate the components. Dropping the concluding direction suppresses the loop but the model still fails to terminate, stalling in the mid-40s. A continuous closed-form solution of the same objective (a ROME-like rank-one update) matches accuracy but rewrites 96% of the weights and flips 103x as many bits.
For model editing, the message is blunt: the agent's natural-language judgment here can be fully formalized, and the formalization wins on accuracy, edit size, and cost. The $312 versus $0.90 gap is really discovery versus application; what is expensive is finding the failure mode and the direction, not solving for the edit.
The Ising form also maps natively onto quantum annealers and QAOA. As a proof of concept, an eight-variable subproblem ran on an IBM Heron processor: depth-2 QAOA sampled the optimum in 110 of 1,024 shots, against 4 for a uniform control.
The paper's own list lands on the soft spots:
Annealing, bifurcation, and the continuous closed form are statistically indistinguishable, which means the per-row problems are easy for classical solvers. The value here is the formulation, not the solver; the quantum experiment shows the mapping works, nothing close to advantage, and the paper says so itself.