0.63M Parameter Verifier Matches 7B Models in Specific Tasks
jm_alexia · x · 2026-08-21
This research investigates the minimum size required for a verifier model by pretraining a series of tiny models (down to 70k parameters). It finds that a model with just 0.63M parameters—trained in about two minutes on a single H100—can achieve verification performance comparable to a 7B model on specific tasks.
Key Findings:
- Countdown Task: The smallest capable verifier scores 0.85 with only 0.63M parameters.
- Faithfulness Task: A 2M parameter model (score 0.83) outperforms Gemini's Zero-shot performance (0.70).
- Scale: 19 models (11 trained from scratch), 3.5B tokens, and 30 GPU-hours were involved.
Task Setup:
- Countdown: A number puzzle verifiable exactly.
- Maze: Shortest-path judgment verifiable exactly.
- Faithfulness: Judging if an answer is supported by its source; trained on human labels and benchmarked against a frontier API model.
The project addresses the performance bottleneck of running verifiers over massive corpora during LLM training, offering a path for rapid verification in multi-step reasoning or agent fleets.
More from Models
- Reddit Tries Recursive Meta-Prompting to Cut Qwen's Thinking Tokens by Half — Rare_Potential_1323 · 2026-08-21
- NVIDIA Details Qwen3.8-2.4T Deployment on GB300, Achieving >4K Tokens/s per GPU — PyTorch · 2026-08-21
- Tutorial: Training a local LLM on a new domain via continued pretraining — funJS · 2026-08-21
- Qwen 3.8 27B two-shots a playable 3D amusement park game in the browser — RandumbRedditor1000 · 2026-08-21
- Zhipu's SAO: single-rollout async RL trains stably for 1,000 steps, beats GRPO — teortaxesTex · 2026-08-21
- Gemini's Safety Filters Too Strict? Rejects Kissing and Roadside Photos — Dry-Sympathy-3182 · 2026-08-21