AI UX auditor passes 4 tests then invents problems: 20/27 findings accurate, 2 false positives
noya1979 · reddit · 2026-11-05
A developer building an AI-powered website auditor tested one UX evaluation prompt across five sites using Playwright screenshots and structured data, yielding 27 findings. Reviewing as a professional UX designer: 20 accurate, 5 partially accurate, 2 incorrect.
The key failure came on the strongest site: the model correctly reported no critical/high issues, then produced four low-priority findings anyway — two false positives, treating intentional progressive disclosure as a flaw and questioning legitimate navigation categories. The audit format pushed the model to find problems even when none existed.
Planned fixes: require concrete evidence of user friction, explicitly allow zero findings, evaluate established patterns in context, distinguish observations from verified problems, and stop reporting preferences as defects. A hybrid evidence pipeline (URL + user-uploaded full-page screenshot) is planned after Playwright screenshots missed dynamically rendered sections.
More from coding & agent
- Grok asked for the best coding harness recommends rival Claude Code — mhdfaran · 2026-11-05
- Agent benchmark scores hinge on container configs: 6-point swing from resource limits alone — krishnan · 2026-11-05
- Revolut cut AI coding inference costs ~90% by self-hosting GLM on 56 B200s instead of Claude — sebkrier · 2026-11-05
- VibeCoding a 3D fishing game in 4 days with Kimi K3.1 Pro and DeepSeek: full workflow breakdown — lxfater · 2026-11-05
- Agent eval design: testing partial updates vs full snapshots with a truck dock case — jylusdev · 2026-11-05
- Sentry founder says AI agent 'junior' handles hardest coding tasks well — zeeg · 2026-11-05