Testing 'code verifies, model generates' on 4B models
retarded_770 · reddit · 2026-08-21
A month-long solo experiment tested if a verification harness could substitute for scale at 4B. The pattern splits requests: model extracts facts -> code verifies them (numbers verbatim, ≥60% word coverage) -> code does arithmetic -> next reasoning stage receives only verified facts.
Results:
- Same weights: Harness achieved 6/16 certified-clean vs 0/16 for one-pass (p≈0.008).
- Third-party Holdout: Harness 5/16, Thinking 2/16, one-pass 1/16 (similar token cost).
- Failure: The strict headline result did not transfer to the holdout (0/16), indicating local overfitting.
Residual Issue: Cross-stage incoherence persists—the model names a constraint correctly in stage 2 but violates it in stage 4. Reasoning-trained weights only partially mitigate this.
More from Research
- Resource: One of the most rigorous math explanations of Transformers — stanfordnlp · 2026-08-21
- CfP: Learning and Reasoning with Graphs Workshop at BNAIC '26 — pbloemesquire · 2026-08-21
- SenseTime releases open-source SenseNova U1.5 with MoT architecture — multimodalart · 2026-08-21
- DeepMind partners with game studio to explore long-term memory and multi-agent economies — GoogleDeepMind · 2026-08-21
- GigaBrain-0.7 Launch: 'System-3' Architecture Tops Robot Leaderboards — 机器之心 · 2026-08-21
- Linear Algebra Textbook 2nd Edition Adds Backpropagation & Attention — prof_g · 2026-08-21