Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligence
Zhiqin Yang, Jingwen Fu, Yuhan Liu, Hengyu Liu, Yonggang Zhang, Kainan Cao, Zizhuo Zhang, Chenxin Li, Ruibin Yuan, Jiahao Pan, Jiankai Sun, Zhenyuan Zhang, Yibo Li, Yunlong Lin, Jing Xiong, Sida Lin, Bo Han, Wei Xue, Yike Guo
cs.AI
2026-09-01
HKUST and collaborators split scaling beyond human supervision into reward provenance and experience generation, then locate remaining human provision on an L0-L4 ladder.
RLVR has a clean story in math and code: answers can be checked, programs can be run, and models such as DeepSeek-R1 grew on that signal. Open-ended writing, long-horizon agents, and tasks with several valid outcomes do not. Human preference labels also fail to keep up once traces get long and tasks get harder than the evaluators.
The bottleneck is which parts of the learning loop still need people to keep supplying material, not a cheaper score. A team led by HKUST treats "scaling beyond human supervision" as two axes: where the reward evidence comes from, and who produces the tasks and environments. A five-level ladder then marks remaining operational human provision. The ladder tracks responsibility during learning. It is not a capability ranking, and it does not erase the human knowledge already baked into pretraining, tools, or environments.
On the reward axis, L0 pairs each instance with an answer or preference. L1 freezes those criteria into a reusable reward model, rubric, or LLM judge. L2 scores rollouts from model confidence, sample agreement, available references, or environment outcomes, while tasks and environments stay mostly external. The experience axis then moves: L3 generates or reorganizes tasks, curricula, and environments during learning; L4 is the idealized loop in which policy, reward, tasks, and environments co-evolve, with humans limited to intent, safety constraints, and independent audits.
Optimizers are not a second ladder. PPO and GRPO are ways to consume a reward: GRPO replaces a critic with group-relative advantages and is the common LRM choice. Internal signals such as sequence surprisal and token entropy can be used as rewards, but neither is a calibrated probability of correctness.
This is a survey, so the result is the map. RLHF typically spans L0 and L1. Rule-checked math and code remain close to L0 because each new item still needs a target. Entropy minimization and consistency rewards sit at L2, and they only help if the base model already puts mass on good solutions. On the experience side, the literature runs from fixed human datasets to proposer-solver loops, executable environments, and self-play.
Each step changes the failure mode. Consensus can amplify a shared mistake. Tests and environment scores can be gamed when the rules are incomplete. Generated tasks can be too easy, too hard, or invalid. Self-play can narrow the experience stream. The paper treats these as first-class risks, not footnotes.
Anyone running RLVR or agent RL can place their pipeline on this map: where the score comes from, where the tasks come from, and which layer still waits on people. The more useful claim is evaluative. Policy capability, feedback fidelity, and experience quality should be measured separately. A gain on a frozen external benchmark does not prove that the learning loop is healthy.
This is a map against self-deception, not a new algorithm. L4 remains conceptual. The path toward superintelligence is written as a possible consequence of co-evolution, not as a result already in hand.
The authors say the ladder is not a set of mutually exclusive classes; methods often sit in transition. It tracks the operational source of supervision during learning, not the historical human knowledge in pretrained models, corpora, tools, or environments. Direct numeric comparison across papers is weak because models, compute, decoding, and verifiers differ. Coverage of safety, multilingual reasoning, multimodal interaction, and long context is selective, and the superintelligence discussion stays conceptual. The title is more aggressive than the body, which states that supervisory autonomy is neither general intelligence nor a guarantee of real capability growth.