Fleet dev's own lesson: 200 prod-derived eval cases still meant always playing catch-up
BraceSproul · x · 2026-10-16
BraceSproul shares his own stumble building Fleet: nearly 200 individual eval cases, all derived from real production issues, meant the team never got ahead of problems and was only playing catch-up — reinforcing his argument that eval work must go beyond prod traces.
More from coding & agent
- Claude counts tokens, not messages: 10 tricks to make your usage limit last all day — Aiden_Tech_Ai · 2026-10-16
- How to convert 20-year-tenure engineers to AI-native development before deciding who to retrain — alex_verem · 2026-10-16
- Gemini 4 Argon goes live in Antigravity with its own weekly quota for Pro subscribers — rseroter · 2026-10-16
- LangChain revamps human-in-the-loop middleware with typed schemas for tool interrupts — Hacubu · 2026-10-16
- Robot fully controlled by frontier LLMs assembles gadgets autonomously for 1.5 hours error-free — ihorbeaver · 2026-10-16
- Why the future of agents is basically Node and JavaScript, in one argument — verrsane · 2026-10-16