FinDendro Benchmark: GPT-5 Answers 61% of Facts It Claims Not to Know
DataHound26 · hf · 2026-11-09
The FinDendro benchmark tests LLM temporal self-knowledge with 2,226 dated US facts (1999–2026) spanning macro series, month-end stock prices, and Fed decisions. Key findings:
- GPT-5 answers 76% of controlled facts correctly but claims to know only 38%, and still answers 61% of facts it says it doesn't know
- Knowledge claims persist up to seven quarters beyond the measured knowledge boundary
- All three common leak safeguards—vendor-stated cutoffs, asking the model, instructing it to answer as of a past date—prove unreliable; a declared date removes most prior-knowledge claims without changing accuracy
Implication: financial backtests inside a model's training window cannot be certified leak-free by self-report; behavioral measurement of knowledge boundaries is needed.
More from Research
- Survey by 30+ authors maps modern NLP tokenization, from subword wins to its flaws — rdesh26 · 2026-11-09
- No 8 GPUs needed: six hands-on steps to actually understand transformers — cneuralnetwork · 2026-11-09
- SwarmEvolve: 450 latent-space agents boost program search 5-13x — aigclink · 2026-11-09
- ByteDance Seed's Hierarchical Residuals decouple residual width from compute width in Transformers — ByteDance-Seed · 2026-11-09
- Paper in Phil Trans B: human-AI agent interactions are bidirectional and reshape who we are — AnnaCiaunica · 2026-11-09
- Even top 3D-gen models keep making the same basic mistakes, researcher notes — teortaxesTex · 2026-11-09