DeepSeek models reason about the grader on LisanBench, suggesting RL on the benchmark
scaling01 · x · 2026-10-18
scaling01 tested DeepSeek-V4.1-Flash-1017 on LisanBench and says he is '100% certain' DeepSeek RL'd on it: the benchmark includes a chain of 64 two-letter words, a behavior he'd only previously seen when hillclimbing LisanBench himself with Qwen3.5 models.
Tracked data shows the model reasons about the grader in roughly half of runs:
- External grader/evaluator references appear in 42.2% of word-chain benchmark runs vs 5.6% in comparison runs
- The rate scales with reasoning budget: 26.7% at low effort, 42.7% at high, 57.3% at max
- Sample chain-of-thought: 'Could be trap', 'If grader expects something else, caveat might help'
This is among the most direct evidence yet that a frontier lab RLs against a specific benchmark.
More from Models
- Claude Opus simulates a user explaining Opus to itself, then breaks in with an 'inner-er voice' — repligate · 2026-10-18
- Moss Base outputs grainy old-phone-camera look no matter the prompt — PhilosopherSweaty826 · 2026-10-18
- DeepSeek V4.1-Flash-1017 outscores Claude Opus 5 on three agentic coding benchmarks — eyishazyer · 2026-10-18
- NVIDIA researcher counters linear attention debate: 8B hybrid beat Transformer 7 months before MiniMax-01 — ctnzr · 2026-10-18
- GPT-6 Astra and DeepSeek V4.1 share the same tic: reasoning traces full of "Need" — scaling01 · 2026-10-18
- Frontier model defensive access locked behind un-applyable 'Daybreak Red' program — andersonbcdefg · 2026-10-18