MNIST-PRO: MNIST is Back as a Partially Observable World for AI Agents
Vernon Toh, Navonil Majumder, Zhengyuan Liu, Nancy F. Chen, Soujanya Poria
cs.AI, cs.CV
2026-09-01
MNIST-PRO turns digit ID into 64x64 glimpse search: Gemini-3.7-Flash falls from 98% full-image to 65%, and a stitched canvas recovers part of the gap.
Multimodal models already answer questions when the full image is in the prompt. Agents in the wild see a slit of the world at a time: they have to choose where to look next and fold those scraps into a working perceptual state. Standard VQA boards never ask for that. Embodied suites mix perception with collisions and low-level control, so a failed episode is hard to attribute.
MNIST-PRO turns digit recognition into a controlled POMDP. Full-image accuracy sits between 83% and 99%, so the ceiling for "can you see a digit" is already high. Difficulty is injected by a 64×64 glimpse, a four-way walk, and a memory constraint.
Each 28×28 MNIST digit is upscaled to 224×224 and binarized. The agent sees a gray canvas with only the current window unmasked. Moves shift that window by 32 pixels; a predict action ends the episode. Level 1 is one digit with a 36-step budget. Level 2 concatenates two digits into a 224×448 canvas with a 78-step budget and requires order. One hundred episodes per setting, ten per class on Level 1.
Four internal memories isolate how state is built. Image Only keeps every past glimpse (H=∞). Textual State sets H=1 and carries only the model's written thoughts. Metric Grid Map also uses H=1, but forces relative coordinates and per-cell features. A Visual Memory Canvas pastes glimpses onto a coordinate-aligned image, either online after every step or offline on a frozen trajectory. A coding harness and cross-episode persistent memory are extra conditions. Ten models are queried, including Gemini, Claude, GPT, Qwen-3.8-27B, and GLM-4.6V.
In native multi-turn with full visual history, Gemini-3.7-Flash scores 75.0%/47.0% on Level 1/2 against 98.0%/97.0% full-image. Claude-5-Sonnet walks the most (15.96/43.24 steps) and scores 23.0%/0.0%. GLM-4.6V answers after 2.76 steps on average and scores 13.0%/0.0%.
Under the controlled Image Only protocol, Gemini-3.1-Pro-Preview drops from 99.0% full-image to 38.0% on Level 1. Gemini-3.7-Flash lands at 65.0%/22.0%. No model clears 30% on Level 2. An audit of 1,600 Level 2 traces finds 323 single-digit answers; 238 of those never touched a pixel of the other digit and used 15 of 78 budgeted steps. Once a wrong identity is written into the textual state, later contradictory glimpses often fail to overwrite it. Exact-sequence accuracy is 26.0% even when both digits are more than 75% covered, and 1.0% when coverage of either digit stays at or below 25%. Seeing the ink is not the same as reading it.
Replaying the same traces onto an offline canvas exposes an interpretation bottleneck. Claude-5-Opus already covers 83.0%/80.5% of strokes under Textual State at 41.0%/13.0% accuracy; the frozen canvas raises that to 76.0%/67.0%. Claude-5-Fable moves from 38.0%/18.0% to 81.0%/69.0%. In the coding harness, Gemini-3.7-Flash stitches its own canvas and reaches 88.0%/63.0%, using about 2–3× the steps of the in-loop Memory Canvas. Persistent memory does not add accuracy (85.0%/62.0%). GPT-5.6-Terra invents an ASCII-canvas procedure and invokes it in about 2% of episodes.
Open models fare worse. GLM-4.6V revisits at over 60% under textual memory, and even a grid map covers only 46.7%/33.1% of the canvas.
The split between "recognize a digit" and "build a perceptual state under partial observability" is now measurable. For GUI agents and active-vision robots, the practical message is narrow: longer visual history, coordinates, and tools help different models differently, and no representation wins across the board. Stopping policy and belief revision are the sharper failure modes. Many traces fail after the evidence is already on screen, or they halt with most of the budget unused.
MNIST is a diagnostic, not a deployment task. It shows that stuffing extra frames into context does not substitute for a writable spatial state.
The canvas is clean binary ink with no texture, lighting, or occlusion. The window, stride, and four-way action space are far from a real camera. One hundred episodes per cell is a thin sample. Several model IDs are unreleased Gemini, Claude, and GPT snapshots, so reproduction is expensive. Persistent memory is reported for two models only. Gains from the offline canvas are described as organization and readout, but the canvas also changes the input format, so it is not a pure same-representation control. Loops in open models are not split into planning error versus instruction following.