antirez: benchmark games trivially guessable from output biases; Astra misses Cannon
antirez · x · 2026-11-04
antirez critiques a paper on models playing games: he likes the direction but argues the chosen games can be trivially identified from output biases alone — e.g. the "green" channel (the player's color) concentrates at the center, a strong hint of a mostly static object.
He finds it shocking that Astra, with full tools, can't recognize the game as Cannon or similar, and suggests testing with games that can't be guessed from output biases.
More from Models
- Kimi K3's compute load too heavy to serve economically, analysis argues — teortaxesTex · 2026-11-04
- Kimi K3.1 Pro leaked doc shows Gated DeepSeek Sparse Attention for long context — zephyr_z9 · 2026-11-04
- Nathan Lambert: Chinese open-model labs have never missed a weights release promise — natolambert · 2026-11-04
- Moonshot releases Kimi K3.1-Pro as open weights on Hugging Face — ApprehensiveAd3629 · 2026-11-04
- Moonshot open-weights Kimi K3.1 Pro: 2.8T-param MoE with 1M context — scaling01 · 2026-11-04
- GLM-5.3 goes into production on T-Systems' sovereign Industrial AI Cloud in Munich — jietang · 2026-11-04