AI builder shipped 510 chars of JS where a customer had 70,000 — a field report on blaming the model vs the pipeline

AdministrativeBad752 · reddit · 2026-10-14

An app platform's engineering lead recounts a production incident: after swapping the builder to GPT-6 Luna on Sept 23, a customer's 70,000-char JavaScript section shrank to 8,000, then 1,400, 874, and 510 across edits, with 37 manual restores before the report landed.

Key findings:

The refine pipeline (plan call + write call with @@REFINEUNCHANGED@@ sentinels) validates whether output is a valid app — not whether it's still the customer's app.
A first diagnosis via Claude Code blamed the new model for returning patches instead of full files; every link was true but the weighting wasn't.
Full production data: 1.4% of refines lost >60% of JS on the old model vs 7.6% on Luna — same hole, 5x less often. Only 13 of 111 losses had the patch shape; 26 were short rewrites published on the first try, 78 went through retry.
The real fix was a deterministic check on the process (flag outputs far below parent size), not reverting models. The fix shipped the day the report arrived.

Original post →

AI builder shipped 510 chars of JS where a customer had 70,000 — a field report on blaming the model vs the pipeline

More from coding & agent