SkillRoll: Testing Agent Skills in Simulated Worlds
Pretend_Conflict8159 · reddit · 2026-08-26
Testing Agent Skills is hard due to dependencies on external state like files or APIs. The author released SkillRoll, an open-source harness that evaluates skills within simulated worlds.
Core Mechanism:
- Input: The task for the skill.
- World: A private description of the environment and reaction logic.
- Success Criteria: Observable success behaviors.
A "Dungeon Master" agent handles tool calls based on the World description, and an LLM judge determines the outcome.
Usage:
- Tests are written in a single Markdown file.
- Run via skillroll init and skillroll eval.
- Tested with openai/gpt-5.6-luna-pro at <$0.01 per eval.
MIT-licensed with no telemetry, it serves as a solid regression testing tool for agents.
More from coding & agent
- Agent Sandbox Usage: Receipt URL Becomes Critical Verification Gate — Common_Dream9420 · 2026-08-26
- MCP solved access, but A2A solves the economics of data — jpolec72 · 2026-08-26
- Coding Agent's worst output: Context compacted — JFPuget · 2026-08-26
- Agent orchestration is real: built a two-agent news system in 100 lines — Positive-Ad3618 · 2026-08-26
- Seeking practical, low-code AI Agent framework for local use — Infamous-Gain-6786 · 2026-08-26
- LLM as a judge: How do you trust the judge? — maylad31 · 2026-08-26