Vicki Boykis: LLM app evals are missing 'walking around the app'
vboykis · x · 2026-09-03
Vicki Boykis argues the current debate over LLM-product evaluation misses a key practice: "walking around the app".
Key points:
- Apps with non-deterministic components should at minimum test online outputs in production daily; offline evals are a bonus if capacity allows.
- Beyond model outputs, touch every part of the product daily: dropdowns, search, toggles, load times, devices, network speeds, payment gateways, onboarding for new users in North America vs New Zealand, CDN image loading.
- Borrowing Dan's phrase, she frames it as a daily, habitual product mindset — broader than transactional QA bolted on at the end of a PR.
Related event: Fable 5.1 Review: Faster and Cheaper, But Coding Not 'Solved'(2 posts)→
More from coding & agent
- JSON prompting with MOSS Image 5B: bbox coordinates for exact multi-room layouts — cocktailpeanut · 2026-10-21
- Rewriting an agent skill: A/B challenge runs show skills should only add what agents don't know — jmschreiber91 · 2026-10-21
- Developer builds himself a 'Rork MoTS' agent on Grok Bot, aiming for full autonomy before GTA 6 — rudrank · 2026-10-21
- OpenAI Live API tip: control voice sessions from your backend via sideband WebSocket — pbbakkum · 2026-10-21
- OpenAI's MultiLangBench: PHP scores highest (91.3%) at building a database engine from scratch — ricklamers · 2026-10-21
- UserProof lets Claude Code watch your user session recordings and boost conversion for $29/mo — doooyle · 2026-10-21