Zero-Weights Python Rule Bot Beats Llama 2 70B on GSM8K

lvwerra · x · 2026-11-19

PyBot, a chatbot with no weights and no training — just 600k lines of Python rules written by a swarm of Codex agents in 1.5 days — scores 15.2% on GSM8K, beating Llama 2 70B (11.6%) and GPT-3.5 davinci-003 (12.6%), 8.6 points behind 2023 ChatGPT (23.8%), all with direct answers and no chain-of-thought.

The rules were iterated over 16 rounds of 6 parallel Codex agents. A browser demo (via Pyodide) shows which module "won" each reply; code is open at github.com/zw2700/pybot-math. A satirical-but-real baseline experiment, and a fun demo of using coding agents to mass-generate rule code instead of model weights.

Original post →

More from Fun

Fun channel →