Engram Talk: Why LLM Training on Private Data Collapses

AI Engineer · youtube · 2026-08-12

Jack Morris delivers a deep dive into the core challenge of integrating enterprise private data with LLMs: Scaling Compute on Context.

The 'Synthetic Data Wall' of Private Data

The Flaw of Direct Training: If you train a model from scratch directly on ten thousand financial reports, the loss drops to near zero, but the model's generation ability completely collapses.
Exhausted Axes: The deep learning revolution relied on three axes: more public data, bigger models, and more compute. When dealing with private corporate data (like internal emails or meetings), the data axis is fixed, and training from scratch is off the table. Compute is the only axis left to push.

Bottlenecks of Current Approaches

KV Compaction: Only processes data already fitting in the context window, entirely skipping gradients.
On-Policy Distillation: Effective, but raises the question of what to distill, as raw documents won't work.
Continued Pretraining on Synthetic Data: Promising, but it overwrites pretraining knowledge and eventually hits a 'synthetic data wall'—once the model absorbs the generated dataset, it stops scaling, lacking pretraining's continuous scaling behavior.

The Ultimate Goal: Self-Play

Morris points out that the ideal solution should possess the self-improvement property of AlphaGo: as the model gets better, the training questions it generates get harder, allowing continued compute investment to buy depth rather than flattening out prematurely.

Original post →

Engram Talk: Why LLM Training on Private Data Collapses

More from Research