CMU Course Syllabus: Systematically Decoding LLM Offline and Online Evaluation
841io · x · 2026-08-09
A CMU researcher shared the detailed syllabus for an upcoming graduate course, Quantitative Evaluation of Language Technologies (11-801).
The course focuses on the scientific evaluation of language models and is structured into two main modules:
- Measurement: Covers measurement modeling, core NLP task metrics, and deep dives into specific metric classes like sequences, rankings, and multi-turn interactions.
- Experimentation: Split into pre-deployment offline experimentation (dataset construction and hypothesis testing) and post-deployment online experimentation (A/B testing).
The curriculum aims to help students critically interrogate existing evaluation methods and lay the groundwork for developing new ones.
More from Research
- Human-Robot Juggling: IROS 2026 Paper Extends Cascade Record 5x — Jan_R_Peters · 2026-08-09
- Best Practices for Continual Pretraining (CPT): A Curated Resource List — liuzhuang1234 · 2026-08-09
- NeurIPS OpenReview Timeline Updates: Does 'Modified' Indicate Rating Changes? — Responsible-Read-138 · 2026-08-09
- Goodfire CTO on Concept Manifold Geometry & a $1000/Month ML Research Agent — The Cognitive Revolution · 2026-08-09
- Goodfire CTO on Concept Manifold Geometry and a $1000/Month ML Research Agent — The Cognitive Revolution · 2026-08-09
- CVPR 2026 Paper Open-Sources PersonaLive for Expressive Portrait Animation in Live Streaming — tom_doerr · 2026-08-09