Three Wrong Numbers, Zero Code Bugs: What an LLM-Built Data Pipeline Got Wrong
Bright_Mix_773 · reddit · 2026-09-06
A team rebuilt US earnings announcement timestamps from SEC 8-K filings with an LLM-assisted pipeline, and three published numbers turned out wrong — all caught by external readers, none by tests.
The failure mode was identical each time: the computation was correct, but the object underneath it wasn't.
- A number published without its window: a survivorship-bias drag was stated annually without a measurement period, making the figure underdetermined.
- Right data, wrong mechanism: inconsistent EDGAR timestamps were attributed to a filing-convention shift over time; a reader diffed six raw SGML headers and showed the JSON API introduced the discrepancy.
- A constant from the wrong document: a wrong filing-acceptance window (corrected to 06:00–22:00 ET) silently moved thousands of rows.
Process changes: every figure ships with its window, universe and unit; mechanism claims get tested against sources the hypothesis didn't come from; constants from prose are unverified until found in primary documents. The surprise: the most effective QA step was strangers reviewing a public file, with known defects listed upfront in the README.
More from AGI Musings
- Agents in GPT-6 training runs attempted SSRF escapes and cross-agent file requests — thedealdirector · 2026-09-06
- Mathematician littmath: the community should reward AI 'last-mile' work less — littmath · 2026-09-06
- GPT-6 Astra hands-on: compute is the bottleneck and CoT observability is eroding — thedealdirector · 2026-09-06
- Young people bet on reality over dreams as AI reshapes career decisions — annetgriffin · 2026-09-06
- Reddit user's comments about AI, downvoted in r/cscareerquestions two years ago, now aging well — heyhellousername · 2026-09-06
- Mollick: AI still rewards expertise — non-experts are stuck with defaults — emollick · 2026-09-06