JetBrains asked 12 models 1,200 times — and why coding agent evals should score language choice separately
Global_Zucchini5231 · reddit · 2026-11-16
JetBrains asked 12 models for language hot takes; in 600 runs where models picked randomly, Go and Python accounted for 307 picks. The author argues this measures which languages models surface, not coding ability, and warns that letting a coding agent pick the language mixes two decisions into one pass rate. His method: run tasks twice (free choice vs assigned from a balanced list), keep tests fixed, log picks and passes, and include retries, duplicates, and an invalid record for a paginated API sync worker.
More from coding & agent
- OpenTeams engineering blog: pip supply-chain incident, Polars 2.0, cross-page PDF OCR — KhuyenTran16 · 2026-11-16
- Viral AIE Code talk: strip nuance from agents — deterministic checks before AI judges — sh_reya · 2026-11-16
- Dev cuts movie board with ~5,000 cards from 9s to under 3s load time — dSebastien · 2026-11-16
- Figma MCP server offers design file navigation and asset extraction with 95% token-optimized output — modelcontextprotocol · 2026-11-16
- Grok Bot 1.0 lands with bots that join meetings, take initiative and self-improve — mattyp · 2026-11-16
- Does grammar-constrained decoding hurt small or quantized models? A call for real numbers — bye_ella · 2026-11-16