0.63M Parameter Verifier Matches 7B Models in Specific Tasks
jm_alexia · x · 2026-08-21
This research investigates the minimum size required for a verifier model by pretraining a series of tiny models (down to 70k parameters). It finds that a model with just 0.63M parameters—trained in about two minutes on a single H100—can achieve verification performance comparable to a 7B model on specific tasks.
Key Findings:
- Countdown Task: The smallest capable verifier scores 0.85 with only 0.63M parameters.
- Faithfulness Task: A 2M parameter model (score 0.83) outperforms Gemini's Zero-shot performance (0.70).
- Scale: 19 models (11 trained from scratch), 3.5B tokens, and 30 GPU-hours were involved.
Task Setup:
- Countdown: A number puzzle verifiable exactly.
- Maze: Shortest-path judgment verifiable exactly.
- Faithfulness: Judging if an answer is supported by its source; trained on human labels and benchmarked against a frontier API model.
The project addresses the performance bottleneck of running verifiers over massive corpora during LLM training, offering a path for rapid verification in multi-step reasoning or agent fleets.
More from Models
- Muse Spark Open runs locally on a Mac M3 Ultra via llama.cpp: frontier model on desktop — pcuenq · 2026-10-16
- OpenAI ships GPT-Live-1.1: new voices in six languages, fewer confirmations — OpenAI · 2026-10-16
- Muse Spark model appears on LMArena, testable in Battle and Agent modes — arena · 2026-10-16
- Running a multi-model UI design arena: Paseo orchestrates four design drafts per model — vista8 · 2026-10-16
- Ilya Sutskever questions Meta license clause taxing MaaS providers above $20M revenue — ilyasu · 2026-10-16
- Muse-Spark open-weights drop: 816B-A42B MoE model gets 359GB MLX 3-bit build — ns123abc · 2026-10-16