HardGeoBench: 93 models tested over 18 months, distillation plus RL hurts general ability
Afinetheorem · x · 2026-10-19
Kevin Bryan (U Toronto) published the leaderboard of HardGeoBench, his geo-location benchmark using never-posted photos with no metadata or obvious clues, testing raw LLMs without search tools (pass@1). Top models nail the exact location (within 10 miles) 57.5% of the time on hard samples; he suspects no model can exceed 75%.
Based on 1.5 years and 93 models, he argues distillation plus quantization plus task-specific RL meaningfully harms general intelligence: the four worst performers are GPT 6 Luna, Mimi 2.6 Pro, and Qwen 3.8 27b, with DeepSeek v4.1 Flash 1017 the worst ever. Best performers include multiple Gemini models (incl. 4 Argon), post-Mythos Anthropic models (steady improvement, no jaggedness), Muse Spark (incl. Open), and Astra; Kimi 2.3 and GLM5.3 are good but still at GPT4.1/Haiku 5.5 level.
More from Models
- Gwern's "Truesight" Directory Collects LLM Stylometry De-anonymization Research — voooooogel · 2026-10-19
- SlopCodeBench: Best AI Coding Agent Solved Just 14.8% of Evolving Task Checkpoints — aj_marshall · 2026-10-19
- Grok's phone-call feature only works in the US via Bland AI, testing reveals — bytebot · 2026-10-19
- Anthropic-trained model insisted 3×4=36 'while possessed': hallucinations from bad labels — lennysan · 2026-10-19
- LobsterAI report: DeepSeek's 97% cache hit rate makes it China's cheapest office agent model — AlchainHust · 2026-10-19
- Nemotron 3.5 Bolt Hands-On: Tiny KV Cache Makes It Ideal for Small Multiuser Local Setups — 1ncehost · 2026-10-19