Google Trends quietly redraws samples daily, shrinking significance in 70% of replicated econ papers
RexDouglass · x · 2026-09-04
On the Allegedly Does Not Replicate podcast, Pitt economist Lester Lusher describes his project "When Samples Shape Significance": his team found 73 papers in top econ journals using Google Trends, re-ran the exact queries 40 times for 24 papers with replication packages. T-statistics shrank 70% of the time (31% on average), with some papers losing significance in nearly every redraw — because Google silently redraws its sample daily. Strongest fragility predictors: researcher degrees of freedom (index transformations, partial series, many queries). No p-hacking needed. The episode covers implications for referees, pre-registration, census samples, and unemployment statistics.
Related event: Resampling study shakes significance of top-journal findings(3 posts)→
More from Research
- FlashRender: Few-step camera-controlled generative rendering via MeanFlow distillation — everex · 2026-09-04
- LatentStream: progressive latent memory evolution for streaming video understanding — Hongyu Qu · 2026-09-04
- Temporal Context Routing aligns script timing in joint audio-video generation — Yichen Liu · 2026-09-04
- Puffin-World scales unified multimodal model with native 3D world states — Kang Liao · 2026-09-04
- Agent reliability degrades with trajectory length, but no benchmark isolates it, dev finds — rio_ARC · 2026-09-04
- The Bayes Bandit: A Mathematical Take on Curiosity in Reinforcement Learning — CatAstro_Piyush · 2026-09-04