willcb: most modern LM ideas are testable on 3B-active models at 32K context

willcb · x · 2026-11-17

Adding to his argument that research ideas should be compressed into smaller experiments, willcb states that fairly few ideas in modern language modeling can't be tested on 3B-active models at 32K context — a concrete budget guideline for compute-limited research teams.

Original post →

More from Research

Research channel →