Astryx Introduces 'Vibe Tests' for Evaluating AI Coding Agents
Vjeux · x · 2026-07-31
Front-end design system Astryx shared their engineering practice for evaluating AI coding agents, called Vibe Tests.
Core methodology:
- Motivation: To objectively measure how friendly different systems (Astryx, Tailwind, shadcn, raw HTML) are for AI-generated code.
- Neutral Evaluation: They abandoned the early, narrow approach of checking strict adherence to their own system. Instead, they use a target-neutral evaluation that scores properties any good code output should have.
- Engineering Loop: By using naive prompts, isolated agents, and neutral rubrics, they run nightly automated tests. Recurring agent failures are then turned into PRs to fix the codebase.
This approach turns the subjective "AI experience" into a quantifiable, nightly automated benchmark.
More from coding & agent
- Study: Over-reliance on Coding Agents Hurts Code Comprehension — omarsar0 · 2026-07-31
- 10-Year Expert Shares How to Build AI Eval Systems — danielmckinn0n · 2026-07-31
- Awesome OpenClaw: Curated List of OpenClaw Ecosystem Resources — tom_doerr · 2026-07-31
- Hermes Agent Open-Source Skills: Buzz Integration and Media Attachments — Teknium · 2026-07-31
- AI Automates Freight Matching in 8 Minutes, Dispatchers Become Supervisors — avz008 · 2026-07-31
- From Basics to Agentic RL: A Deep Dive into LLM Tool Use — natolambert · 2026-07-31