CogEvol: Towards Efficient and Reliable Learning Environment Generation
Shangqing Tu, Daniel Zhang-Li, Yucheng Wang, Shiyu Gan, Yanpeng Wang, Huiqiang Rong, Mofei Chen, Shen Yang, Yini Chen, Yinuo Duan, Haoxuan Li, Binglin Liu, Ye He, Danqi Zheng, Zhanxin Hao, Yuxuan Wu, Mengting Tao, Yuqiu Liu, Jifan Yu, Juanzi Li, Bin Xu, Lei Hou, Huiqin Liu, Yu Zhang
cs.CL, cs.AI
2026-08-31
CogEvol turns a course brief into slides or interactive HTML in one pass; the 27B model scores 63.7 on HTML-500 with zero hard fails, median 17s/59s in production.
Classroom materials are becoming executable: renderer-valid JSON slides, and self-contained interactive HTML (simulations, games, code playgrounds). General coding agents can produce them, slowly. A direct-API editing loop in this paper averaged 152 seconds; published systems often spend 200 to 600 seconds per edit. Looks are not playability. A slide with invented field names hard-fails the production renderer. An HTML page can be visually dense with dead buttons. Flagship coding models are also too large and too expensive for this workload at classroom scale.
CogEvol names the job Learning Environment Generation: one brief in, one finished artifact out, no multi-turn agent.
Post-training only. CogEvol-4B starts from dense Qwen3.5-4B; CogEvol-27B from hybrid Qwen3.8-27B (48 gated-delta-net layers plus 16 full-attention layers). Three stages: mix SFT teaches the two output contracts, slide RL teaches composition, HTML RL teaches playability.
The 53,687 SFT conversations are production-grounded. Slide teachers pass parse, schema, production render, and a multimodal judge. HTML mining exported 119,122 production pages, found a 21.7% hard-fail rate under a Chromium probe, then regenerated and re-executed. SFT reaches a 99.2% slide parse rate and does not raise interactive quality; HTML scores fall as SFT steps increase.
RL is GRPO. Slide reward is 0.6 VLM fidelity plus 0.4 geometric rules (collisions, overflow, fake charts). HTML reward splits visual quality, content, dual viewports, and an interactivity probe, with hard failures zeroed. The incident: an old screenshot-only reward left games at 18.8 once rescored; after a always-on probe and hard-fail gate, games reach 57.6. The sentence they keep: interactivity must be measured, not judged.
At inference, scaffold editing retrieves from about a million historical widgets and emits component-level patches, cutting output tokens and latency by about 76% on an internal set. MAIC-UI click-to-locate plus unified diffs averages 6.3 seconds per edit on the same backbone, versus 151.7 seconds for a direct API, about 23×. Serving was ported to Ascend 910 A3 at application-level parity with A800 (500/500 valid parses); missing speculative decoding and stack gaps still leave a throughput hole.
Over 220k production requests, median latency is 17 seconds per slide (P95 26) and 59 seconds per interactive page (P95 107). On internal suites, CogEvol-27B scores 63.7 on HTML-500 and 83.7 on slide-std. Claude Opus 4.8 leads HTML at 67.2 but hard-fails 19 of 500 pages; 27B fails zero. GPT-5.4 scores 66.0 HTML with 13 hard fails. Qwen3.8-Max ties 83.7 on slides given the full 34KB spec, then scores 35.3 on HTML with 204 of 500 pages dead. Human testing: unusable pages 25% (6/24) to 10% (3/30), pages that cannot be entered 2/24 to 0/30.
Open CogEvol-4B: 61.7 HTML-500, 75.1 slide-std. The serial recipe pays a forgetting tax: 27B slides 84.8 to 83.7, all on layout. Suites stay internal. The HTML scorer is the training reward, run by a Qwen3.8-family 27B VLM.
This is a rare vertical post-training report that includes production traffic and a disclosed reward-hacking incident. For education generation, or any UI that must actually click, the transferable piece is the reward: a screenshot judge will breed pages that look finished and do not run; an executable probe is what reverses it. 27B is about 26.9× smaller than GLM-5 in total parameters; at their public list prices the API is 15-22× cheaper per artifact than Opus or GPT-5.4, and 4B is Apache 2.0.
Because the suites are held in, flagship rows are not a public leaderboard. Claude still leads HTML overall. 27B wins on zero hard fails and the slide contract, not on every sub-type.
There is no standalone limitations section; the caveats are already in the body. Topics and the scorer are unpublished, so external models submit an API and contamination is hard to audit. The HTML judge shares a family with Qwen3.8-Max. Human tests are n=24 versus 30 with prompts that are not identical. Language mixing and element stacking survive both rounds because the reward does not price them. The HTML corpus is 69.8% simulations and contains no 3D, so SFT bias is left for RL to patch. On Ascend, prefix cache and speculative decoding fail silently because GDN state cannot be truncated like a KV cache; application parity is not serving parity (46.1 vs 167.4 RPM on production A800).