Benchmark critique: shadow-puppet score favors edge alignment, zero of 6 models solve a plate
cloneofsimo · x · 2026-11-06
cloneofsimo argues a shadow-puppet benchmark has questionable design choices: its scoring formula (55% IoU + 45% outline F, plus hole/wrist/RULA terms and peek penalties) heavily favors edge alignment over human-perceptual similarity, and the model never sees its score — so why hide it?
He notes the 'blob' baseline isn't random: it's a convex cover that systematically aligns many edges, and by any reasonable perceptual metric the model outputs aren't better. Even the blogpost concedes a move is almost as likely to worsen the shadow as improve it, and none of the six evaluated models replicated a single plate — the best solved less than one.
More from Research
- SteerAct lifts robot action-model success ~10% with just 24 rollouts per task — pliang279 · 2026-11-13
- PhD thesis explores intra-turn interaction: AI interfaces beyond turn-based dialogue — OfirPress · 2026-11-13
- CMU trains memory writing with RL: Qwen3.5-4B rivals GPT-5.4 on continual learning — aviral_kumar2 · 2026-11-13
- NTU paper: decision info lives in middle layers, last six layers removable loss-free — Minh-Anh Nguyen · 2026-11-13
- Inkling 2 Small's width-4 sconv forces a tiny rolling 3-token conv cache per layer at inference — andrew_n_carr · 2026-11-13
- Open-source legal conflict detector runs on encoder models only, 133-clause contracts in under 20s — ladyofcode · 2026-11-13