The next AI vendor lock-in won't be the model API, it'll be the harness
Arindam_1729 · x · 2026-11-11
It started with OpenAI's Logan Kilpatrick asking why his company's model was the only one not tested in its native harness in a benchmark. Arindam expands:
- If a model's score depends heavily on which harness it runs in, the benchmark is no longer testing the model — it's testing the model plus the tools, prompts, and workflow it was tuned for
- Every lab now builds both the model and the place you use it, improving them together
- The cost: switching gets harder. Move a model out of its home harness and you may lose quality you were paying for
Conclusion: the next vendor lock-in in AI won't just be the model API — it'll be the harness the model works best in.
More from coding & agent
- KAIST's WikiTrace: evidence-linked memory fixes agent "memory-induced degradation" — dair_ai · 2026-11-11
- Why inference speed will be AI products' biggest competitive edge in 1–2 years — itsOmSarraf_ · 2026-11-11
- What happened when I moved vector retrieval into Snowflake SQL instead of orchestrating RAG outside — dima806_dima · 2026-11-11
- OpenAI engineer runs Codex for 26 days straight hunting Atlantis, spawning "vibe archaeology" — Polymarket · 2026-11-11
- Edward Hughes: hand-built AI harnesses have a ~3-month shelf life — Machine Learning Street Talk · 2026-11-11
- He replaced a $5,000/month ghostwriter with 11 Claude Skills and earned $47,000 from one post — aryanXmahajan · 2026-11-11