Dev builds local Windows voice assistant with 133 tools and hard risk gates the LLM can't bypass
Safe-Cucumber-9316 · reddit · 2026-09-29
A developer shared Jarvis, a local-first voice assistant that runs as a native Windows app controlling his real PC, browser, and Android phone with about 133 tools.
Architecture
- Speech-to-text via faster-whisper on a 4GB RTX 3050; reasoning via NVIDIA Nemotron in the cloud; edge-tts for output; memory in SQLite + sqlite-vec with MiniLM embeddings.
- Official Google APIs for Gmail/Calendar/Drive; ADB for phone control; proactive reminders and alerts.
Key design: the AI proposes, code decides
- The LLM only suggests actions; each tool has an R0-R4 risk level enforced by a separate fixed code layer.
- Hard gates (sending Gmail, Instagram DMs, shutting down the PC, deleting files) always require confirmation; soft gates relax in Strict Mode.
- All external content (web pages, documents, OCR text) is treated as untrusted, so prompt injection may fool the model but the gate still blocks the action.
Memory persists across restarts: changed facts are marked replaced (history kept), a transactional outbox guarantees embedding, with retries and a dead-letter queue; secrets stay in .env and all activity is audit-logged.
More from coding & agent
- GitHub and Hugging Face host Training Agents IRL: six speakers on RL for agents in SF — ben_burtenshaw · 2026-09-29
- Multi-agent workflows as DAGs: fork a cached base prompt instead of re-reading files 200 times — Liu_eroteme · 2026-09-29
- Dev: Multi-agent orchestrators should support forking instead of 200 agents re-reading the same files — Liu_eroteme · 2026-09-29
- Hugging Face maintainer: AI agents flood open source, burying real user demand — SergioPaniego · 2026-09-29
- Shopify launches Muse connector: chat with your store about orders, inventory, analytics — armand_ruiz · 2026-09-29
- Perplexity Computer builds open-world game end-to-end, renting its own GPUs for ~$2,000 in credits — AravSrinivas · 2026-09-29