Agent's rule held on 120 CIKs, broke on 494: lessons in LLM-in-the-loop verification

Bright_Mix_773 · reddit · 2026-09-06

A team maintaining an open SEC 8-K earnings dataset (64,938 filings, 808 S&P 500 companies) let an agent chase a timestamp inconsistency. It sampled 120 CIKs and produced a clean rule—only dormant filers lacked times—which they published to four repos before a full check across 40,000 filings and 494 companies falsified it: 13 active filers including HCA, Ford and Foot Locker broke the rule, in a direction that would exclude companies from needed checks.

Key takeaways:

Fourth wrong claim published in three days; none were coding errors—every calculation was correct, but the object each sentence claimed to describe was wrong
The recurring failure: hypotheses confirmed against the same evidence that generated them, and agents happily report a pass; the check must come from a source the hypothesis never touched (494 companies via a different query path; earlier, six hand-downloaded SGML headers beat 64,938 rows of their own aggregate)
Other three errors: a survivorship figure published without its measurement window, an era-level explanation that was actually an API artifact, and a constant pulled from the wrong document
The process change that paid: known defects go in the README label, not a footnote—publicly listing published errors drew three of four corrections from strangers auditing the file

The author notes they haven't measured whether this generalizes beyond one dataset.

Original post →

Agent's rule held on 120 CIKs, broke on 494: lessons in LLM-in-the-loop verification

More from coding & agent