Simon Willison Releases smevals: A Lightweight Eval Suite for LLMs
Simon Willison · rss · 2026-08-01
Simon Willison, working with Prime Radiant lab, has released smevals, a lightweight eval suite designed to evaluate the capabilities of different models, prompts, and harnesses.
The tool is built for AI coding agents: developers can simply instruct their agent to run uvx smevals docs to learn the tool and build an eval suite. It decouples runs from grading operations, uses YAML files for checks, and includes a local web server or static HTML builder for exploring results.
More from coding & agent
- Developer Demos Vibe Coding Workflow: Build and Ship Apps in 30 Minutes — vista8 · 2026-08-01
- Dev Showcases Local Multimodal AI Companion App Running on RTX 4060 Ti — aziib · 2026-08-01
- Conductor in Action: Orchestrating Autonomous Agents from a Single Workspace — charlieholtz · 2026-08-01
- Cisco AI Launches Antares Quickstart Guide for 15-Minute Code Vulnerability Localization — aminkarbasi · 2026-08-01
- Beyond One-Shot Prompts: The Real Value of AI Agents Lies in Underlying Context — thisiskp_ · 2026-08-01
- Developer Builds Automated Music Discovery Tool Using ChatGPT Sites Plugin — simpsoka · 2026-08-01