Stanford's Priced Guidance lower-bounds whether LLMs can forecast future research via paid hints
stanfordnlp · x · 2026-10-07
Percy Liang shares a new Stanford method: naively sampling whether an LLM can generate a future paper is intractable (astronomically small tail probability).
Priced Guidance instead has the model (Astra) play "20 Questions" with a guide LLM holding the paper, charging each hint by surprisal to precisely account for bits transferred — yielding a lower bound on P(LLM forecasts a future paper's idea). It doubles as a new eval for frontier models.
Related event: Stanford's Priced Guidance Measures LLM Novelty(3 posts)→
More from Models
- Is Bel's math edge scale or synthetic data? TeortaxesTex bets on data — teortaxesTex · 2026-10-07
- Fable claims its new release equals roughly five OpenAI Navier-Stokes-level results — willdepue · 2026-10-07
- Reddit user ships 'surgical abliterated' 27B red-team model with zero refusals — Least_Dog_8556 · 2026-10-07
- OpenAI Researcher Surprised AI Lab Math Results So Far All Hold Up — willdepue · 2026-10-07
- OpenAI dots losing to Meta's Muse surprises AI community — BLUECOW009 · 2026-10-07
- Inception launches Mercury Decide on OpenRouter: free structured-decision model doing 14 decisions/sec — StefanoErmon · 2026-10-07