Pamela Fox swaps GPT-5.5, Kimi K2.6, DeepSeek V4 and Mistral through the same app scenarios
AI Engineer · youtube · 2026-10-19
AI Engineer published a workshop from the AI Engineer World's Fair 2026 in which Microsoft Principal Cloud Advocate Pamela Fox runs identical scenarios against GPT-5.5, Kimi K2.6, DeepSeek V4 Flash, and Mistral Large 3 on Microsoft Foundry, arguing benchmarks won't tell you how a model behaves in your app — swapping it in will.
- Capability gaps: starts with a letter-counting prompt and works up through RAG groundedness, image input, multilingual pitfalls, tool calling, tool calling in a loop, and a LangChain v1 travel agent; prompt engineering closes some gaps but not image understanding.
- Evals: string-check evals and LLM-as-a-judge at scale; Microsoft's new ASSERT tool generates test sets from a spec; DSPy prompt optimization finds a prompt even the cheapest model satisfies.
Full chapter list included; ASSERT is open-sourced at responsibleai/ASSERT.
More from coding & agent
- Claude cut a 2.6-hour video export to 20 minutes — but only after being told to rebuild the server — byhewar · 2026-10-23
- 3 flight-booking rules my AI agent checks that I'd never do by hand — Efistoffeles · 2026-10-23
- OpenJet claims Qwen3.8-27B coding runs 1.9x faster than llama.cpp on one RTX 3090 — JeremyCMorgan · 2026-10-23
- Building an autonomous SEO agent in Hermes with three skills: programmatic pages, Keywords Everywhere, DataForSEO — EXM7777 · 2026-10-23
- Building a sports betting AI agent with onchain execution and human approval by default: architecture and lessons — Dry-Comfortable-2678 · 2026-10-23
- How do you audit AI agent decisions in regulated workflows? A field survey — Ok_Way_1993 · 2026-10-23