Scaling Video Generation for Reasoning: At What Cost?
Weihang Guo, Xiaoyu Wu, Yifei Wang, Niloofar Mireshghallah, Lydia E. Kavraki
cs.CV, cs.AI, cs.LG
2026-09-29
On a controlled Rubik's Cube video benchmark, lower MSE does not track hidden-state reasoning; symbolic state supervision lifts a 20M model to 550M-level accuracy at 1/30 compute.
Video generative models are increasingly used as world models: given an initial frame and an action program, predict how the environment changes. The failure is quiet. A video can look perfectly plausible while contradicting the actions that came before it. This paper drags that failure into the open with a controlled task: a solved 2×2×2 Rubik's Cube, one fixed camera covering three faces, nine prescribed turns, and the model must generate the whole video. Stickers rotate onto hidden faces and reappear later, so predicting which colors show up requires inferring how each intervening action changed the hidden state.
The design settles both evaluation and data supply. Initial state plus action sequence uniquely determines every future state, so the simulator provides exact ground truth with no human or model-based judging. Action sequences can be sampled without limit, which let the authors train seven model sizes from 20M to 1B parameters, autoregressive (AR) and bidirectional (Bidir) video diffusion Transformers, one epoch, every training video used exactly once.
The most informative intervention is symbolic state supervision: the 20M video model additionally predicts all 24 sticker-color distributions, supervised with cross-entropy, and the predicted distributions (after stop-gradient) are fed back to condition a separate video pass. The total loss is Lflow + 0.1·Lstate on the same 3M videos in the same order as the baseline. Both variants supervise all 24 stickers; Front12, which feeds back only the 12 visible distributions, already captures the full gain.
| Setting | Metric | Result |
| 70M AR, 0.1 PF-days | Frame accuracy | 44.6% |
| 1B AR, same compute point | Frame accuracy | 0.3% |
| 1B AR, 3.14 PF-days | Frame accuracy | 83.7% |
| 1B vs 70M, 1.5M videos each | MSE / frame accuracy | 1B has lower MSE; accuracy 3.67% vs 41.00% |
| 120M AR, 5M→8M videos | Frame accuracy | 48.11%→49.33% |
| 20M + state supervision, 3M videos | Frame accuracy | 31.1%→67.3% (0.0445 PF-days) |
| 550M video-only | Frame accuracy | 67.1% (1.33 PF-days) |
Three contrasts matter more than the table. The 270M model draws correct cube geometry after 0.034 PF-days but first gets all 12 stickers right only at 0.57 PF-days, a 17× gap: looking right and computing right are different skills. Moved into discrete vector space, a 2.8M-parameter Transformer hits 99.86% full-trajectory accuracy (bidirectional, 20-action sequences) for 4×10⁻⁴ PF-days; state tracking itself is easy, carrying it through video generation is not. Exposing all six faces lifts AR mean frame accuracy from 34.67% to 93.44%, so partial observability is where the difficulty concentrates.
Treat the power-law extrapolations with care: fitting the middle versus the final third of the compute frontier moves the projected compute for 98% accuracy by orders of magnitude (4.2×10⁵ versus 23.55 PF-days for AR frame accuracy). The paper labels these illustrative projections, not budget forecasts.
For teams building world models or robot video prediction, this is a cheap, reproducible diagnostic protocol: fixed task, exact ground truth, unlimited data, no frontier models needed. Two findings travel. Validation loss cannot compare reasoning across models; the 1B model has lower MSE than the 70M while being an order of magnitude worse on state accuracy, so selection needs direct state evaluation. And explicit state supervision gets a 20M model to 67.3% for 0.0445 PF-days while the 550M video-only model needs 1.33 PF-days to reach 67.1%, a 30× gap that says scaling is not the only lever. The "correct turn, wrong colors" silent failure is exactly the kind that breaks downstream control.
This is a methodological contribution, not a capability jump. The 1B model does reach 83.7% eventually. Scaling works; it is just expensive.
The authors state that the handcrafted sticker representation may not generalize and that the power-law fits are illustrative only. On top of that: a single synthetic environment with one renderer, and no transfer to real video is tested. Symbolic supervision assumes the state structure (24 stickers) is known; where that signal comes from in real tasks is unanswered. Frame accuracy requires all 12 stickers correct, a strict criterion, and the partial-credit picture lives in sticker accuracy. Single-epoch training keeps the scaling curves clean but departs from multi-epoch practice.