Evals in 2026 101: one taxonomy for LLM apps, agents, and training/inference systems
WolframRvnwlf · x · 2026-10-31
Shared as "how to do evals in 2026 101," this thread splits "pipeline evals" into three distinct problems:
1) Multi-step LLM apps: evaluate end-to-end outcomes plus every nondeterministic step; instrument functions with W&B Weave Ops/Calls and run versioned evals over the same cases and scorers.
2) Agents: additionally evaluate trajectories, tool use, handoffs, and final state; Agent Lens traces agents, clusters production failures into test cases, and supports human-calibrated custom judges.
3) Training/inference systems: beyond quality, test throughput, latency, reliability, and cost under realistic load — what CoreWeave ARENA is built for.
A clean three-way taxonomy for engineers building eval systems.
More from coding & agent
- Engineer rates 10 kinds of work on 4 automation levels: only PM stays human — kieranklaassen · 2026-10-31
- Against Skill sprawl: why 50 saved prompts beat 50 installed Skills — TruthTellerTom · 2026-10-31
- Rewriting 5 agents as decision workflows: 20 seconds, 2 cents, 19/20 correct — generativist · 2026-10-31
- You May Not Need an Agent: Decision Workflows Did 20 Docs in 20s for 2 Cents — generativist · 2026-10-31
- Softi: a shared cloud computer where you and AI agents work side by side, with 24/7 automations — Small-Let-3937 · 2026-10-31
- Developer adds an AI assistant to GridCraft that runs app commands from plain English — dfinke · 2026-10-31