In-context learning is becoming continual learning: frontier LLMs ingest novel domains without fine-tuning
Ryoiki-Tokuiten · reddit · 2026-11-21
A Reddit long-post argues that in-context learning in frontier LLMs is severely underestimated: feed a curated 150k-token context (a compressed representation of task rules and output distribution) and the model fully ingests a completely novel domain and works with it as if trained on it — no fine-tuning.
Two firsthand examples:
- A highly niche math area with almost no materials anywhere; the model produced new results that normally take the author 2-3 weeks
- BEND-2, a brand-new programming language; the model generated meaningful laws and proof files despite never having seen it
The author also tested a continual-learning loop: read curated context → work → produce output → update context → repeat. Updated contexts measurably improved in rule adherence and consistency, though models still occasionally make wrong assumptions. Are these glimpses of continual learning?
More from AGI Musings
- Academic says AI has redrawn the domain of human achievement; "not in my discipline" is likely wrong — soumitrashukla9 · 2026-11-21
- Engineer's rant: mathematicians stop at "a recipe exists" and never taste the cake — gerardsans · 2026-11-21
- Wharton MBA deck adopts 'Technology Bottlenecks' paper: same task, 23 vs 83 pressure by job — soumitrashukla9 · 2026-11-21
- 'Humans built this': from LSTM to Transformer in 29 years, no credit to the machines — teortaxesTex · 2026-11-21
- Kevin Hartnett's new book "The Proof in the Code" charts Lean's history transforming math and AI — ipeirotis · 2026-11-21
- Gary Marcus recorded his Anthropic IPO-skeptic podcast days before the $83/share offering — GaryMarcus · 2026-11-21