Jose Valim shows Campfire AI benchmark was rigged: Elixir faced far stricter checks than Go/Rust
zeeg · x · 2026-10-12
Elixir creator Jose Valim dug into why AI-generated speedup PRs were rejected for the Elixir port of Campfire while Go/Express/Rust ports got accepted — and found each project had different starting points, prompts, and verification bars. Elixir was the only port required to preserve pinned Rails behavior and the only one forced to keep Redis with queue draining, actively enforced. His comparison gist shows Go passing only 12/214 desktop screens (network 0/198), Rust with a strict harness but no committed parity report, Express with just 26 browser assertions, while Elixir passes all 9 contracts and 65 gates with a validator that refuses to benchmark unverified source. His takeaway: the people telling you not to read code can't even set up a fair AI benchmark — because that requires reading the code.
More from coding & agent
- CC Switcher: open-source Windows app to run multiple Claude Desktop profiles in parallel — zeeg · 2026-10-12
- Guard Lab: open-source 3D game where an LLM guards a safe and you try to rob it — flowersslop · 2026-10-12
- Microsoft Open-Sources IQ Solution Accelerator Unifying Work, Foundry and Fabric IQ — adnan_hashmi · 2026-10-12
- Cohere launches North 2: rebuilt agent harness, memory, skills, org-wide cost caps — dl_weekly · 2026-10-12
- Claude Cowork one-time setup: a 3-stage guide to stop wasting time and tokens — alex_verem · 2026-10-12
- Vibe coding's hidden ownership costs mean SaaS isn't going anywhere — MarcJSchmidt · 2026-10-12