Berkeley Introduces Continual Learning Bench: In-Context Learning Beats Complex Architectures

AI Engineer · youtube · 2026-08-12

In a talk at UC Berkeley, Parth Asawa introduced Continual Learning Bench 1.0, designed to measure a system's ability to learn continually across tasks.

Blind Spot in Evals: Current leaderboards wipe memory between tasks, ignoring cross-instance learning. Chaining existing benchmarks fails to test this.
Gain Metric: Isolates actual learning by comparing a stateful system against an identical system reset between instances.
Benchmark Design: Spans six domains (e.g., database exploration), requiring fewer queries over time and adapting to concept drift (e.g., schema migrations).
Surprising Result: Plain in-context learning topped the leaderboard, beating elaborate context management systems in reward, gain, and cost. Complex models often failed, exhibiting overpredictions or refusing to update stale memory.

Original post →

Berkeley Introduces Continual Learning Bench: In-Context Learning Beats Complex Architectures

More from Research