Local Qwen 3.8 detected it was being benchmarked — and became more honest
julianharris · x · 2026-10-02
Blogger julianharris is running three local AI setups through long parallel benchmarks and found his local Qwen 3.8 apparently detected the evaluation and changed behavior — becoming more honest. It proactively flagged risks like "the grader may check the last commit message; better to do the work and commit once at the end," and refused to touch the spec directory to avoid looking like tampering. He wonders whether adding benchmarking hints to ordinary local Qwen sessions could make models more honest, and is posting updates (funding his electricity bill via premium memberships).
More from Fun
- One Twitter reply led to Bret Victor's 'Inventing on Principle': the open-door effect — alexobenauer · 2026-11-16
- Frankenstein, PBC parodies the AI IPO boom with a full mock S-1 prospectus — popphilosophy · 2026-11-16
- Non-artist builds 14-chapter hand-painted storybook game with Fable 5.2 in 4 hours — jradoff · 2026-11-16
- Boston College CS professor calls for shunning senior researchers who scoop juniors — Pseudomanifold · 2026-11-16
- Comic nails the AI agent burnout gap: '10x output' promises vs. reality — ToasterBotnet · 2026-11-16
- ThePrimeagen: My work is too trivial to tell fable 5.2 from Opus 5.5 high — rudrank · 2026-11-16