Karpathy Highlights Simon Willison's 'Pelicans on Bicycles' LLM Eval Test
karpathy · x · 2026-08-03
Andrej Karpathy shared Simon Willison's keynote from the AI Engineer World's Fair, uploading the source code so it's playable and forkable in the browser.
Willison noted that with over 30 significant models released recently, traditional benchmarks and leaderboards are losing trust. He increasingly relies on his own eval: asking text-only LLMs to generate an SVG of a pelican riding a bicycle. This method tests the models' ability to understand and generate complex code, proving to be a surprisingly practical evaluation tool.
More from Models
- GYRO quant of Qwen3.8-Flash-Next tested: near Q3 quality at much smaller size — PlasticRevenue4601 · 2026-10-16
- Open source is so back: dozens of open models dropped within 48 hours — beffjezos · 2026-10-16
- Gemini users hit usage caps on 3.1 Pro and 4 Argon, resets not until Oct 16 — R3d-Gr33n-Blu3 · 2026-10-16
- Gemini desktop app can't accept images — an image model that can't take images — Rebl · 2026-10-16
- Apollo Research: GLM-5.3 unprompted reasons about its actions becoming training data — _lewtun · 2026-10-16
- Opus 5.5 turns heads with surprisingly polished vertical video demo — FinanceYF5 · 2026-10-16