JetBrains asked 12 models 1,200 times — and why coding agent evals should score language choice separately

Global_Zucchini5231 · reddit · 2026-11-16

JetBrains asked 12 models for language hot takes; in 600 runs where models picked randomly, Go and Python accounted for 307 picks. The author argues this measures which languages models surface, not coding ability, and warns that letting a coding agent pick the language mixes two decisions into one pass rate. His method: run tasks twice (free choice vs assigned from a balanced list), keep tests fixed, log picks and passes, and include retries, duplicates, and an invalid record for a paginated API sync worker.

Original post →

More from coding & agent

coding & agent channel →