LiteLLM SDK adds agent evaluations: one call scores pass rate, cost, tokens, duration
ycombinator · x · 2026-11-18
LiteLLM launched agent evaluations in its SDK for developers who want to compare models and harnesses on their own tasks.
Previously you had to write the eval loop, spin up temporary workspaces, and collect results in a spreadsheet. Now you can reuse your agent, define typed tasks with a success check, and call litellm.agenteval() to get a scorecard with pass rate, cost, tokens and duration — the sample card compares GPT-6.2 Sol ($0.0003, 6.26s) vs Gemini 4 Hydrogen ($0.0005, 7.54s) on a fixadd task.
More from coding & agent
- zeeg vows to keep overengineering until he's 'accidentally built an email client' — zeeg · 2026-11-18
- Dev pairs Runway live avatars with Nous Hermes Agent, will open source it — alexcovo_eth · 2026-11-18
- 4 launches, zero virality — but 150+ OHAN invites and 2 VC calls — prasanna_says · 2026-11-18
- Dev pairs Runway live avatars with Nous Hermes Agent to replace dot-and-blob agent UIs, plans to open source — alexcovo_eth · 2026-11-18
- Microsoft's Agent Framework PyPI Package Hacked to Steal Azure and GitHub Credentials — Kyrannio · 2026-11-18
- Grok-powered agent given $50 and a 'survive or shut down' challenge grew it to $5,763 in 48 hours — nikola_mr64990 · 2026-11-18