CoRT: Optimizing LLM Fine-Grained Credit Assignment via Counterfactual Replay
_akhaliq · x · 2026-07-31
The paper introduces CoRT (Counterfactual Replay) to address the mismatch in rubric-guided LLM RL where rich criterion-level feedback is collapsed into a single response-level advantage and broadcast uniformly to all tokens.
By replaying responses with and without rubric criteria, CoRT turns policy-internal log-probability changes into normalized token-level credit weights without training an auxiliary scorer, changing the reward, or requiring extra generation/verifier calls. Across multiple models and instruction-following benchmarks, CoRT improves matched response-level GRPO by 4.4 points on average while remaining competitive with learned token-relevance methods.
Related event: ByteDance Proposes CoRT to Optimize LLM RL Credit Assignment(2 posts)→
More from Research
- Goodfire's Nick Cammarata argues InceptionV1 polysemanticity wasn't caused by superposition — petrusenko_max · 2026-10-16
- LLM ideation experiment: just ask for non-obvious ideas — and Opus beats smaller models by a wide margin — arjunrajlab · 2026-10-16
- SimpleMark: multi-bit LLM watermark bounds per-token distortion with f-divergence budget — rishabh16_ · 2026-10-16
- Stanford paper: generative representation learning cuts tokenizer training compute by 7-8x — HirokatuKataoka · 2026-10-16
- Cohere Labs to present confidence estimation for reasoning models at EMNLP 2026 — Cohere_Labs · 2026-10-16
- Vals RSI Index extrapolates Google models hitting 130% autonomous AI R&D by Oct 2027 — HaydnBelfield · 2026-10-16