Physical Intelligence pits zero-shot LLM Fable against fine-tuned π0.5 on real robots

paigeinsf of SE3 Labs ran a systematic real-robot comparison between the zero-shot LLM-driven Fable 5.2 and the fine-tuned VLA model π0.5. The zero-shot model clearly led on memory and long-horizon tasks—areas where LLMs excel—and the author later clarified the boundaries of the conclusions while open-sourcing the experiment dataset.

Confirmed

Setup: 10 episodes per task, scored by milestones with partial progress counting; π0.5 was fine-tuned on 500 teleoperation demonstrations (dataset se3labs/real-robodojo-tel…, open-sourced).
Memory and long-horizon tasks (blocks hidden under identical cups, requiring sequential uncovering): Fable averaged 64 points vs π0.5's 7.
Fable won on fine manipulation: charger insertion 37 to 9, tube insertion 26 to 14.5, though absolute scores were low for both.
Packing was the exception: fine-tuned π0.5 scored 51 (consistently 45–60 per round), while zero-shot Fable swung wildly (5–1…).

Boundaries of the Conclusions (Author's Clarification)

π0.5 needs data from the specific workstation to run; its scores reflect the 500 collected demonstrations, and more targeted data could push its scores higher.
π0.5 is an older open-weights model; newer π-series models may be stronger.

Why It Matters

This is a rare real-hardware head-to-head between a zero-shot LLM-driven robot and a fine-tuned VLA, rather than simulation, with open data and setup for reproducibility.
The results show LLM memory and long-horizon reasoning can translate into robotic task advantages, challenging the common assumption that workstation-specific fine-tuning is mandatory; but absolute scores on fine tasks remain low, still far from practical use.

2026-11-12 ~ 2026-11-12 · 5 related posts

Primary sources

Zero-shot LLM Fable beats fine-tuned pi0.5 VLA on memory tasks 64 vs 7 — paigeinsf · 2026-11-12 08:30 (UTC+08:00)
Physical Intelligence clarifies: zero-shot LLM already beats fine-tuned open VLA — paigeinsf · 2026-11-12 08:31 (UTC+08:00)
SE3 Labs releases 500 bimanual teleop demos behind its real-robot benchmark — paigeinsf · 2026-11-12 08:31 (UTC+08:00)

[source] Zero-shot LLM Fable beats fine-tuned pi0.5 VLA on memory tasks 64 vs 7 — paigeinsf · 2026-11-12
Real-robot eval: fine-tuned π0.5 wins box packing 51-39, zero-shot Fable dominates memory tasks — paigeinsf · 2026-11-12
LLM-driven arm beats π0.5 on precision tasks 37 vs 9, but loses on box packing — paigeinsf · 2026-11-12
[source] Physical Intelligence clarifies: zero-shot LLM already beats fine-tuned open VLA — paigeinsf · 2026-11-12
[source] SE3 Labs releases 500 bimanual teleop demos behind its real-robot benchmark — paigeinsf · 2026-11-12