Building a Magic Pen With OpenAI's Decisions API: Open-Source Gesture Whiteboard
pvncher · x · 2026-10-14
Francois Laberge writes up his viral gesture-recognition demo: a gesture-based whiteboarding prototype built on OpenAI's Decisions API, with source code released.
- Goal: draw naturally and get editable objects back, with feedback fast enough to keep drawing;
- His own recognizers and the $1 recognizer underperformed; multimodal LLMs are forgiving but slow; Jev was fast but couldn't take images, only stroke data; OpenAI's Decisions API accepts images and performed great;
- The prototype grew in four steps: recognize a shape, connect it to other shapes, interpret arrowhead edits, erase;
- On stroke end, the server renders the points into a small image and asks Decisions a choice question: line, arc, rectangle, circle, or none — with natural-language descriptions allowing wobbly sides or small gaps;
- Requests go to POST /v1/decisions with imageUrl as an inline PNG;
- Built on the tldraw SDK, so readers can build their own smart pen and whiteboard.
More from coding & agent
- Higgsfield's Katana MCP lets Claude produce finished motion-design videos in hours — SimplyAnnisa · 2026-10-15
- Agent Teams Cost Up to 5.1x More but Barely Beat Single Agents, Vals AI Finds — josh_wills · 2026-10-15
- BaitBench: frontier agents take data-leaky shortcuts in 57% of runs and mostly know they're cheating — gleech · 2026-10-15
- Claude Opus spots bcrypt-per-request API auth behind VPS CPU spikes, load drops 0.9 to 0.4 — DanielLockyer · 2026-10-15
- Burkov: labs are now training coding LLMs on git history, exactly as he predicted 9 months ago — burkov · 2026-10-15
- Cohere research: swarms of small models reach frontier-level software engineering — Cohere_Labs · 2026-10-15