Researcher building benchmarks to tell whether LLMs compute or recall answers
paraschopra · x · 2026-10-27
lossfunk Batch 9 resident Adya Srivastava studies why LLMs with identical accuracy behave very differently on unfamiliar problems: did the model compute the answer or recall it? Her three-part approach: benchmarks testing reasoning robustness beyond accuracy (rewrites that shouldn't change answers, false values planted mid-solution, three independent probes per problem), mechanistic interpretability on open-weight models, and findings guiding post-training and inference-time interventions. Long-term goal: a reliability layer for agents that triggers verification loops and allocates reasoning compute selectively. Early results span 5 models and 3 task types.
More from Companies & People
- "Is it time to fork Zed?" Community pushback over the editor's AI direction — gnukeith · 2026-10-27
- Cal.com co-founder and CEO Peer Richelsen to appear on The Arena podcast today — JosephJacks_ · 2026-10-27
- Jason on Microsoft's 60K Buyout Offer: Headcounts Will Freeze as AI Productivity Surges — danilobzdok · 2026-10-27
- AI designs lipid nanoparticles to deliver RNA drugs into solid tumors — paraschopra · 2026-10-27
- Building an open benchmark to compare AI approaches to analog chip design — paraschopra · 2026-10-27
- HackerRank eng lead builds Chorus to study how AI agent swarms cooperate — paraschopra · 2026-10-27