10 open-source GitHub repos to test, eval and red-team your AI agents
victor_explore · x · 2026-11-21
A curated thread lists 10 open-source GitHub repos for testing AI agents (stars checked Oct 2026), covering prompt/agent testing, RAG scoring, coding benchmarks and red-teaming:
- Promptfoo (28k stars, MIT): test prompts and agents from config, LLM evals plus red-teaming/vuln scanning, compare GPT, Claude, Gemini, DeepSeek
- DeepEval (20k stars): unit tests for LLM apps
- SWE-bench (6k stars): real GitHub issues as coding benchmark
- Harbor (6k stars): run agent benchmarks in parallel, test agents like Claude Code and Codex CLI, build your own
- Giskard (6k stars): red-teaming and test generation for agentic systems
- TruLens (4k stars): score each step with LLM judges
- Inspect (3k stars): the UK AI Security Institute's LLM eval framework
- tau2-bench (2k stars): tool-agent-user interaction vs simulated customers
- Autoevals (1k stars): ready-made scorers
- Scenario (1k stars): simulated users test your agent
More from coding & agent
- AI first cut generates 5 variants for in-app audition inside Final Cut Pro — Technovangelist · 2026-11-21
- Dev builds an Android port of Hex entirely from his phone, no laptop needed — johnlindquist · 2026-11-21
- Local DeepSeek via Ollama + FCP MCP auto-cuts a 35-min video in ~4 minutes — Technovangelist · 2026-11-21
- OpenClaw hosts Agent Templates Focus Week with community co-building — heyneighbor · 2026-11-21
- LlamaIndex benchmark: Gemini 4 Neon leads tables and visual grounding at 6.2¢/page — llama_index · 2026-11-21
- Dev pegs median timeline for AI replacing programmers at 2032 — jessi_cata · 2026-11-21