Dev hooks OpenAI Realtime API to a DIY robot arm: sim-trained skills plus vision for tic-tac-toe
Tarun122 · reddit · 2026-10-23
Reddit user Tarun122 connected the OpenAI Realtime API (gpt-realtime-2.1 over WebRTC) to a small DIY robot arm he built, so it can be controlled purely by voice. In the video it plays tic-tac-toe, puts a cube in a bowl, picks its favorite toy car, and hands a cube back.
How it works
- The voice model gets only a handful of tools: look, reach, grab, drop, gripper, park, and stop. Reach and grab aren't hand-coded moves — they're policies trained in simulation, so the model decides what to do and the trained skills decide how
- Fresh images from both cameras (top view and wrist) are injected into the conversation every user turn, letting the model read the tic-tac-toe board itself
- A vision model places a point on the target object in the top camera image, and the reach/grab policy moves to that point
A textbook hobby-grade embodied setup: VLM brain + sim-trained low-level skills, with a reproducible architecture.
More from coding & agent
- THU-KEG Releases RIDAR to Detect and Intervene in Research Agent Integrity Violations — THU-KEG · 2026-10-23
- Same LoRA on every Qwen3.5 size: 9B beats 27B, and nearly all the gain is learning to say no — spilldahill · 2026-10-23
- 200 concurrent LLM agents on a 1GB, $6/month VM — the bottleneck was our own save locks — swarmagent · 2026-10-23
- Phone farms with robotic arms: a new setup lets AI agents bypass app anti-bot controls — TinfoilTricorn · 2026-10-23
- Agent Orgs Win on Latency: ~4x Faster Than Budget-Forced Single Agent — rosstaylor90 · 2026-10-23
- 4 days and nearly 4 Opus weekly limits: a multiplayer A-10 flight sim built entirely by agents — siimsiim · 2026-10-23