Paper: Sol 5.6 Achieves 46% Accuracy on BrowseComp-Plus Without Search
beirmug · x · 2026-08-28
A paper by Sahel Sharify et al., titled "Projecting BrowseComp-Plus onto ClimbMix," examines the realism of agentic search benchmarks.
Background:
- The original BrowseComp-Plus benchmark relies on a fixed corpus of 100K documents curated for specific queries, lacking the noise of real-world data.
- The authors project these questions onto NVIDIA ClimbMix (400B tokens, 553M documents), a real-world general corpus, using a pipeline that decomposes questions into atomic reasoning hops and verifies evidence.
Key Findings:
- Data indicates that the Sol 5.6 model achieves a 46.1% native accuracy on BrowseComp-Plus using parametric knowledge alone (without search tools).
- After projection onto ClimbMix, retrieval difficulty increases significantly, causing the strongest agent's score to drop by 5 points, better reflecting real-world challenges.
More from Research
- Goodfire's Nick Cammarata: neural networks are "already clean" in a possible interpretability breakthrough — tszzl · 2026-10-15
- tszzl: interpretability breakthrough feels like my understanding of neural nets jumped five years — tszzl · 2026-10-15
- LightSuite: fast cell detection and whole-brain atlas registration on a standard workstation — hisspikeness · 2026-10-15
- Pathway CSO pitches BDH as post-Transformer architecture uniting memory, reasoning, adaptation — schwentker · 2026-10-15
- AI-designed molecular scissors cut Alzheimer's proteins — but is it really AI? — gerardsans · 2026-10-15
- EmbodiedEvo tops RoboDojo at 38.95% success by letting robots inherit failed practice — thetripathi58 · 2026-10-15