UIUC's FAS benchmark trains LLMs to predict future research directions, +10.6% FAS
shangbinfeng · x · 2026-10-25
UIUC researchers (Heng Wang, Jiawei Han, Heng Ji et al.) reframe LLM research-proposal evaluation as time-sliced scientific forecasting in an EMNLP 2026 Findings paper. Proposals are scored by whether they anticipate directions appearing in papers published after a cutoff, via a Future Alignment Score (FAS) computed with retrieval + LLM semantic scoring. They build a time-consistent dataset (21,835 paper occurrences, 3,642 instances) and synthesize reasoning traces for gap identification. Fine-tuned Llama-3.1 and Qwen2.5 models improve FAS by up to +10.6%, and two model-generated proposals were implemented with a code agent.
More from Research
- KTransformers-FineTune: CPU-GPU co-compute boosts MoE fine-tuning up to 21.2x — jiqizhixin · 2026-10-25
- State of AI Report 2026: the frontier is now a three-lab race between Anthropic, OpenAI and Google — nathanbenaich · 2026-10-25
- Benchmarks should penalize misalignment, not ambiguity — environment fidelity over prompt over-specification — nirmal_dist · 2026-10-25
- Math is the fastest-growing field on arXiv, at 2x the expected trend — IgorCarron · 2026-10-25
- Nine checkable math puzzles from human to ASI level; Grok passed solved ones but mistook a float estimate for a proof — jarekduda · 2026-10-25
- Git pushes quadruple to 320M per quarter as vulnerability counts lag — LLMs said to boost code security — max_paperclips · 2026-10-25