KierstonBench simulates Allentown, PA to test LLMs on crime investigation—only frontier models hold up

Andrew Semenov · hf · 2026-11-04

Kierston & KierstonBench provide a simulated city-scale environment based on Allentown, PA, evaluating LLMs on criminal investigation and surveillance tasks using linked surveillance records.

• The benchmark chains scattered surveillance records into investigation workflows, offering a reproducible city-scale simulation plus evaluation suite.
• Key finding: only frontier models perform reliably on these tasks, and even they remain vulnerable to deception.
• The work quantifies both the capability ceiling and fragility of LLMs in law-enforcement-style surveillance scenarios.

Original post →

More from Research

• TUM Paper: MTA-OPSD Picks a Different Teacher for Every Token, Boosting Math Reasoning — TUM · 2026-11-04
• Should Garrabrant induction replace Bayesian epistemology? AI safety researchers debate — davidmanheim · 2026-11-04
• LeCun et al. Paper: LLMs Over-Compress at the Cost of Semantic Richness, Explaining Opacity — ziv_ravid · 2026-11-04
• Deep Principle open-sources Hydrogen-1.0, a science model trained on failed lab experiments, scoring 93% on GPQA Diamond — alex_verem · 2026-11-04
• Exactly solvable theory shows when Muon beats Adam: steep spectra, hard targets — CatAstro_Piyush · 2026-11-04
• UC Berkeley's open-source LEANN shrinks a 201GB vector index to 6GB for laptop RAG — thisguyknowsai · 2026-11-04

Research channel →