ByteDance's CoRT: Token-Level Credit Assignment for GRPO Optimization

ByteDance · hf · 2026-07-30

ByteDance introduces CoRT (Counterfactual Replay), a method to optimize Rubric-based GRPO reinforcement learning pipelines.

• The Problem: In GRPO-style pipelines, structured judgments are reduced to a scalar reward broadcast uniformly across all generated tokens, lacking a mechanism for within-response credit allocation.
• The Method: CoRT uses counterfactual replay to rescore the same sampled response under both rubric-conditioned and criteria-free prompts. It uses tokenwise log-likelihood contrasts as a proxy to redistribute the GRPO advantage across tokens.
• Results: Without introducing an auxiliary scorer, CoRT outperforms standard response-level GRPO in the vast majority of comparisons with an average gain of 4.4 percentage points, retaining the simplicity and stability of GRPO.

Original post →

More from Research

• Study: Agentic Harnesses Significantly Improve LLM Compositional Generalization — lateinteraction · 2026-07-30
• Stanford's Free CS336 Course: Building LLMs from Scratch to Become an AI Architect — techNmak · 2026-07-30
• Handroid: A Reconfigurable Robot Shifting Between Humanoid and Dexterous Hand — carlosdponx · 2026-07-30
• RATs: Multi-Agent Robot Lifelong Skill Learning Without Gradients or RL — rsasaki0109 · 2026-07-30
• Cisco's Open-Source Security Model Antares-3B Nears GPT-5.5 at Fraction of Cost — joshua_saxe · 2026-07-30
• DeepMind's Andrew Trask: LLMs Are Like Libraries Without an Index — iamtrask · 2026-07-30

Research channel →