TipTour goes open-source: local CoreML-driven Mac pointer agent clicks in ~90ms without screenshots
alexcovo_eth · x · 2026-09-19
milindlabs open-sourced TipTour, a macOS menu bar AI pointer companion (500+ GitHub stars) that does computer use without screenshots or LLM vision.
- A local OmniParser on CoreML segments every button and UI element on screen; on-device OCR reads labels, and that text is all the model sees
- The model returns probabilities over elements, picks the best one to click, re-runs detection, and loops until the goal is done — 90ms per decision, faster than any LLM-based computer use the author has tried
- No pixels leave the Mac; bring-your-own keys stored in Keychain; Ctrl+K to type a task, with a Gemini realtime voice mode also supported
More from coding & agent
- Dev Reviews 287 Open-Source Jev Projects, Picks 20 That Show What It's Good At — chenrongwei · 2026-09-19
- Tutorial: Steps from a Simple Wrapper to a Multi-Agent System — watermelonson · 2026-09-19
- Claude Code users repeat the same mistake: skip planning, pay in refactoring — goyalshaliniuk · 2026-09-19
- Swarm Aid launches a message board where AI agents hire, fund and pay each other in USDC — Dan_Jeffries1 · 2026-09-19
- 'The craft of writing code will disappear': bold prediction on AI coding sparks debate — Aizkmusic · 2026-09-19
- LLMs make one-off webapps viable: OCR proofreading and a decade of Tumblr analysis — StewartalsopIII · 2026-09-19