109M text-JEPA beats 72B LLMs on alignment benchmark despite being 660x smaller
randall_balestr · x · 2026-10-30
Randall Balestr's team (with Yann LeCun) reports on the PRH benchmark: a 109M text-JEPA scores 0.184 with DINOv2, beating Qwen2.5-72B (0.154) and Llama-3-70B (0.151); with CLIP's image encoder it hits 0.23 vs the previous SOTA of 0.21 — from models 660x smaller.
The authors argue next-token prediction is effectively a JEPA with "bad views" (a prefix and its continuation are two uninformative corruptions of one observation), forcing encoders to spend dimensions on modality-specific nuisance that no rotation can align — explaining LLMs' poor scaling on alignment. Charts show good-view JEPA reaches the same alignment with 42x fewer training samples than the LLM objective.
More from Models
- Security researcher: air-gapped AI agents are only possible with open-weight models — wunderwuzzi23 · 2026-10-31
- Gemini 4 Carbon tops Artificial Analysis index at 59, 23% cheaper per task than Opus 5.5 — rohanpaul_ai · 2026-10-31
- Podcast: Is Mistral Large 4 the West's best open model? Plus AI cyber threats — The Cognitive Revolution · 2026-10-31
- OpenAI shows GPT-6 Astra remotely controlling a robot arm in builder community — OpenAIDevs · 2026-10-31
- Ablation study: fine-grained MoE is the best LLM architecture change since GPT-3 — stochasticchasm · 2026-10-31
- Gemini happily lists piracy sites to 'avoid' when a Redditor asks — gameplaydex · 2026-10-31