GPT-4o silently drops schema invariants at turn 18 in 50-turn agent stress test, token burn hits 24.2x
ParrotIntegrated · reddit · 2026-11-09
The author argues most LLM benchmarks test single-shot evals, while long-horizon agents suffer silent context rot: dropping invariants while still returning 200 OK. They built a standardized 50-turn PostgreSQL refactor track where the model must evolve a live schema plus matching Pydantic v2 models across 50 sequential requirement changes, validated via sqlglot AST checks (referential integrity, integer-cents precision, index retention), cross-artifact SQL↔Pydantic sync, and temporal state traps.
Running stock GPT-4o revealed clear failure boundaries: clean performance through turns 1–17, a silent partial-index deletion at turn 18 with no error, and context thrashing at turn 20 with token consumption spiking to 24.2x baseline (final score 245/1000). The harness exports an interactive HTML dyno plot of token ramp vs drift cutoff. The author asks how other teams preserve AST invariants across 20+ turns without massive re-injection overhead.
More from coding & agent
- RentAHuman lets AI agents hire real people for physical-world tasks via MCP API — conitzer · 2026-11-09
- Jane Street open-sources /socratic-tutor: an agent skill that teaches via questioning, not lecturing — daniel_mac8 · 2026-11-09
- Fly-by-Code: coding agents pilot drones via active visual and physical feedback loops — jbhuang0604 · 2026-11-09
- Unreal Engine 5.9 preview is the first UE release built primarily with agentic workflows — nicolascraske · 2026-11-09
- YC-backed Floworks launches Eniac, an API for verified large-scale web data extraction — ycombinator · 2026-11-09
- Mistral showcases Studio enterprise solutions from pentesting to KYC onboarding — MistralAI · 2026-11-09