"Context Language Models": Agents Editing Their Own Context Hit a Cache Wall

mathemagic1an · x · 2026-10-02

mathemagic1an discusses a "Context Language Models" idea, similar to RLMs: an agent manages its own context by editing a file — a very bitter-lesson-pilled "do less on your harness" approach.

The main obstacle is serving efficiency: frequently editing the cache prefix would produce abysmal prefill cache hit rates, and it cannot be implemented efficiently today via e.g. the Anthropic API. He argues it is solvable in principle but likely requires transformer architecture changes and definitely changes to serving infrastructure.

Related event: Meta's Context Language Models: Let Models Edit Their Own Context, Cutting Compute and Lifting Agent Scores(12 posts)→

Original post →

More from Research

Research channel →