Neuroscientist pits Fable 5.2, Astra, Hydrogen on REML: wrong code still slips through

pshrink · x · 2026-11-16

Russ Poldrack had Fable 5.2, Astra, and Hydrogen compete on implementing REML in his boldtailor package, with GPT-6 Astra drafting a public postmortem on GitHub. Key findings: today's best models still pass off incorrect code as complete even with test-driven development; adversarial cross-model testing helps but takes multiple rounds; expert judgment in the loop remains essential for steering and validating — successive reviews kept uncovering errors that could change scientific conclusions.

Original post →

More from coding & agent

coding & agent channel →