LLMs' mental map of SF tested: best street-level score 0.32 vs 0.23 random
Evening_Hawk_7470 · reddit · 2026-11-13
The open sf-from-memory project had 6 LLMs classify 41,526 San Francisco coordinates from memory (no maps, no web), scored against OpenStreetMap. Key findings: best models hit 86% overall, but a dumb 'not water, not park' baseline already gets 61%; coastline, Golden Gate Park and the Presidio are placed fine, yet street grids are mostly wrong — best Marina street score was 0.32 vs 0.23 for random shuffle. A 5x pricier model tied cheaper ones; high reasoning effort didn't help; one model rendered Marina as near-solid buildings (3% streets vs 21% actual). Author's takeaway for agent builders: facts about real places (is a shop open, which number reaches the front desk) must come from live lookups, never from weights. Next step: testing local models on the Marina block.
More from coding & agent
- Benchmarks won't move users off Claude Code — Antigravity needs real product uplift, dev argues — Justin_Halford_ · 2026-11-13
- Why AI labs must own meeting software: spoken context is the missing fuel for agents — signulll · 2026-11-13
- Didit launches MCP server letting AI agents run KYC, KYB and fraud checks — bosmeny · 2026-11-13
- Open-source project lets a coding agent rebuild any hometown to explore — zsakib_ · 2026-11-13
- mcbench-lite open-sourced: agents build in Minecraft, runs locally on your keys — adonis_singh · 2026-11-13
- Wharton PhD's Claude trading agents up 24.5% vs S&P's 10.69%, $52M copy trades — alejandroll10 · 2026-11-13