HardGeoBench: top AI models now pass@1 above 50% locating metadata-free photos
Afinetheorem · x · 2026-11-20
HardGeoBench, created by Kevin Bryan (U Toronto / All Day TA), tests frontier models at photo geolocation using never-posted-online photos with no metadata and typically no text or road markers—run as a "raw LLM" benchmark with no web search or tools.
Key results:
- The top model's pass@1 has now crossed 50%; 81% of photos have been nailed exactly (≤10 miles) by at least one model, and 94% within 100 miles.
- On May 8, 2025, the three best models ran twice each: 57.5% of images were located exactly at least once.
- Bryan suspects no model can beat 75%—the benchmark's charm is that "if you are 'too' good given relative hardness, you cheated."
- Example: a model identified a photo of an apartment door as Le Corbusier's Cité Radieuse in Marseille, France, off by just 0.02 miles.
More from Models
- Screenshot shows Claude Opus 4.5 launched Nov 24, 2025 — scaling01 · 2026-11-20
- Nov 24 marked on calendar as '1 year intelligence explosion anniversary' — scaling01 · 2026-11-20
- Abundant: Claude safety guardrails flagged RSI tasks but zero refusals actually blocked agents — Madisonkanna · 2026-11-20
- GPT 6.2 Astra tops Kevin Bryan's HardGeoBench as best OpenAI model yet, with caveats — Afinetheorem · 2026-11-20
- GPT-6.2 Sol hits 8.7 min per task, user bets ultrafast tier reaches $100 plan soon — haider1 · 2026-11-20
- Gemini freezes the browser for minutes on long math derivations — jasondeanlee · 2026-11-20