Zero-Weights Python Rule Bot Beats Llama 2 70B on GSM8K
lvwerra · x · 2026-11-19
PyBot, a chatbot with no weights and no training — just 600k lines of Python rules written by a swarm of Codex agents in 1.5 days — scores 15.2% on GSM8K, beating Llama 2 70B (11.6%) and GPT-3.5 davinci-003 (12.6%), 8.6 points behind 2023 ChatGPT (23.8%), all with direct answers and no chain-of-thought.
The rules were iterated over 16 rounds of 6 parallel Codex agents. A browser demo (via Pyodide) shows which module "won" each reply; code is open at github.com/zw2700/pybot-math. A satirical-but-real baseline experiment, and a fun demo of using coding agents to mass-generate rule code instead of model weights.
More from Fun
- The nerfed-benchmark conspiracy joke making rounds in AI circles — AI_Andrew · 2026-11-19
- ChatGPT's nerdy pun: 'Gradient ascent for you, gradient descent for the model' — burny_tech · 2026-11-19
- Cognition's Devin bot haggles a user down to $38 and demands promo tweets for more credits — xiaohu · 2026-11-19
- We Got GTA VI Before ASI — djcows · 2026-11-19
- Dev plans to hike with a laptop in his backpack to train models, AI tells him he's cooked — burny_tech · 2026-11-19
- AI-generated family video goes hilariously wrong: mom tucking insides back in — _onionesque · 2026-11-19