AI-generated code passes all tests but is riddled with opaque contracts, finds Stanford professor's review

RobbWiller · x · 2026-11-17

Russ Poldrack reviewed code Astra & Fable generated for his boldtailor project: all 257 tests pass, but he found terrible internal naming (a write() that writes nothing), widespread unannounced in-place mutation, overused custom context managers with hidden closeout requirements, and a god-like mutable State object with implicit step dependencies. A follow-up AI audit produced a 218-line survey of these opaque contracts, published on GitHub.

Original post →

More from coding & agent

coding & agent channel →