Hallucinated model name sat in codebase for months, caught 6 days before a deprecated model shutdown
Trout_dev · reddit · 2026-10-11
A developer caught a completely made-up model name (gemini-3.6-flash) in their gateway code just 6 days before the entire model family's Oct 16 shutdown. It passed all 130 tests and built cleanly in Docker, because model names are just strings — no type checking or import-time validation, nothing fails until a real API call in production.
Other postmortems in the thread:
- OAuth revocation bug: the disconnect command deleted the MongoDB record and returned "unlinked" without ever calling gmail.revoke(), leaving the Google token live; the right test makes a real API call with the old token and asserts a 401.
- Qdrant sparse vectors: BM25 sparse search (needed since ticker symbols embed poorly) can't be added in place to an existing collection; a full export, recreate, and re-upsert fixed it, retrieval eval MRR hit 1.0.
- text-embedding-004 returned 404 on the live API despite being in the docs — memory writes would have silently stored nothing.
Takeaway: the gap between "passes all tests" and "works when strangers rely on it" keeps being wider than expected.
More from coding & agent
- One question to your coding agent: 6x speedup and 8x cost cut — venturetwins · 2026-10-12
- LangChain launches Managed Deep Agents: agents as directories, fully hosted runtime — EdenEmarco177 · 2026-10-12
- Web Browsing Agent Runs Locally on a 9B Model, Tasks Done in ~11-17s — BangsFactory · 2026-10-12
- 'Vibe coding' should be retired — AI now writes more coherent code than most engineers — signulll · 2026-10-12
- AI scans satellite photos to find driveways and auto-sends $90 powerwash offers — nikitabier · 2026-10-12
- Sentry CEO: startups gluing OTel onto ClickHouse will be the next o11y failures — zeeg · 2026-10-12