Dev asks how to eval an LLM visual reviewer that isn't improving game sprite animation quality
Ok-Author-27 · reddit · 2026-10-29
A developer uses OpenAI's Decisions API (gpt-6-luna) as an advisory reviewer inside a game-character agent: for walk/run cycles it receives 16/12 ordered still frames, an identity reference, and frame durations as JSON, and votes clear/defect/unverified on anatomy, identity/scale, loop-seam continuity, foot alternation, and foot sliding, with an uncalibrated 0.8 cutoff.
The review isn't improving animation quality much, so before swapping models they want to isolate whether the problem is the evidence, the rubric, or the reviewer. They ask: how to build a small human-labeled held-out test set (missed defects vs. false rejections), whether ordered full frames beat contact sheets and targeted frame pairs around foot contact and loop seams, and whether a fast triage classifier plus an expert reviewer that names exact frames would actually improve the next iteration. They solicit first-experiment advice and references for evaluating agentic visual reviewers.
More from coding & agent
- Veteran engineer: juggling 17 projects with agents, deep thinking is a muscle I'm losing — rudrank · 2026-10-29
- Agent-built component flips cash-flow: licensor pays company NOT to open-source it — mattpocockuk · 2026-10-29
- Agent sandbox boat.dev sees record growth day with 100k-view demo and nonstop Stripe orders — zack_overflow · 2026-10-29
- Dev picks Cursor's $60/mo Grok plan as best budget coding combo — pramodk73 · 2026-10-29
- Claude Duet research preview lands in users' hands as Anthropic quietly expands access — RileyRalmuto · 2026-10-29
- LangChain Core 1.7.1 Ships Two Fixes for Streams and Mermaid Diagrams — caolonghua · 2026-10-29