LLMs zero-shot beat fine-tuned world action models on 7 of 10 robot tasks
paigeinsf · x · 2026-11-19
A benchmark comparing zero-shot LLM (Fable 5.2) vs fine-tuned world action model (Dream-1) across 10 robot tasks:
- Dream-1 leads only 3 physical routines: packing objects 54.5 vs 39.0, stacking toy 35.0 vs 31.0, aligning blocks with a ruler 41.0 vs 40.0 — and picks up the ruler in every run.
- Fable wins the other 7, with huge gaps in memory and reasoning: 64 vs 9 on which cup hides the block, 62.5 vs 10 on category sorting, 23 vs 5 on press-by-number. Overall: Fable 38.5 vs Dream-1 22.0.
- Dream-1's top failure mode is going for the wrong target (wrong bin/peg/outlet/button count) in 45 of 100 runs.
Takeaway: LLMs won't replace robot models anytime soon — the best robot brain will likely combine both.
More from Embodied
- SE3 Labs Releases A Wider Dream Eval, Opens 500-Episode Robot Demo Dataset on Hugging Face — paigeinsf · 2026-11-19
- Memory and Reasoning Remain LLM Territory: Fable Beats Dream-1 64 to 9 on Cup Task — paigeinsf · 2026-11-19
- The Secret Wasn't a Bigger Model: 500 Randomized Demos Lift Dream-1 Score 159% — paigeinsf · 2026-11-19
- Fine-tuned World Action Model Dream-1 Jumps 159%, Beats Frontier LLMs on Robot Tasks — paigeinsf · 2026-11-19
- Fine-tuned Dream-1 world action model jumps 159% on real-robot benchmark at 1/100th the cost — paigeinsf · 2026-11-19
- Robotics founder: "sim2real is BS" — labs buying data is a flash in the pan — Scobleizer · 2026-11-19