CritICL: Inference-Time Weak-to-Strong Generalization from Small Language Model Failure Modes
Yufan Wu, Yinghui He, Zhengyi Hu, Lang Wei, Ruichen Li, Qifan Yang, Ting Zhu
cs.CL
2026-08-28
Failure modes stay stable across scales in a family. CritICL retrieves small-model mistake critiques as ICL. Qwen2.5-72B static scores 59.2 vs Consistency@5 at 59.0, one pass.
The usual way to buy math accuracy at inference is to sample many times and vote, to self-reflect, or to call a judge. Those methods score higher and also multiply the token bill by 3 to 7. Another line asks a weak model to coach a strong one online, which still means extra generation per query. A team at Ohio State and Princeton starts from a cheaper fact: inside one model family, a 1.5B and a 72B make the same kinds of mistakes, and the relative frequencies stay stable. Wrong answers from small models are a reusable list of "do not think this way."
CritICL turns that list into an offline critique bank and inserts retrieved critiques as in-context examples. Target weights never move. The static variant generates once per question.
Build CritBank first. Small models in the family (Qwen2.5 1.5B/3B/7B, or Llama-3.2 1B/3B plus Llama-3.1-8B) each produce five chain-of-thought answers on about 15k GSM8K and MATH training items. Keep the wrong ones. gpt-4o-mini then writes up to five failure-mode labels and a natural-language critique per miss, and similar labels are clustered. Each bank record is a question, a wrong answer, labels, and a critique.
Two retrieval policies. Dynamic asks the target model which pitfalls the new question is likely to hit (up to five), then pulls at most five matching critiques; that is two generations. Static ignores the question, aggregates small-model failure frequencies into a family profile, and always retrieves critiques for the dominant pits; that is one generation. Retrieval is by failure mode, not by question embedding similarity.
Qwen2.5-72B-Instruct, Pass@1:
| Method | GSM8K | MATH | Overall | Generations |
| 5-shot fixed | 93.8 | 80.5 | 56.3 | 1 |
| Consistency@5 | 95.0 | 83.2 | 59.0 | 5 |
| CritICL-static | 95.4 | 84.0 | 59.2 | 1 |
On 32B, static overall is 49.8 against Consistency@7 at 49.5. On Llama-3.1-70B, static is 53.1 against Consistency@5 at 51.3. Token cost on MATH with Qwen2.5-32B: static 3768 total tokens, Consistency@5 4814, Self-Reflection 7533. Prompts get longer because of critiques; completions often get shorter (296 vs 308 for 5-shot).
Ablations pin the extra gain on failure-mode alignment. Same five slots and the same retrieval shell: correct exemplars 57.4 average, generic GPT critique 57.7, shuffled labels 57.8, full static 59.9. Aggregate weak-model profiles correlate with 72B failure ranks at Spearman 0.91 inside the family, about 0.46 across families. The MATH lift over the strongest baseline is significant (p=0.018); GSM8K is not (p=0.083). AMC23 and AIME have tiny sets and wide intervals, so tenths of a point there are not a ranking.
This moves weak-to-strong generalization from training onto inference at a cost you can amortize: build the bank once, reuse it for every query in the family. Anyone already running 5-shot can swap "correct worked examples" for "wrong answers plus critiques" and collect most of the gain without Consistency@k. It fits math and scientific QA with recurring pits. It will not replace RL or long chain-of-thought training. Cross-family and cross-domain banks still help, same-family and same-domain help more.
There is no standalone limitations section; the tables do the work. Against the best multi-sample baseline, 72B overall is only 0.2 higher. The real win is fewer generations, not a blowout. AIME-scale deltas are not statistically supported. Offline CritBank needs small-model sampling plus a frontier annotator; Cohen's κ between gpt-4o-mini and humans is about 0.74, so labels are noisy. Dynamic adds a failure-mode prediction call. Weights are frozen, so the lift is ICL-sized. On GPQA in the appendix, static averages 74.4 against Consistency@5 at 71.8, but the main claims still rest on math.