Hacker Project: Running Local LLMs on an Echo Dot 2 with Only 512MB RAM
alberto_zurini · reddit · 2026-08-07
A developer successfully deployed a fully offline voice assistant pipeline on an Amazon Echo Dot 2—equipped with only an ARMv7 processor and 512MB of RAM—achieving local speech recognition and LLM inference.
Core Technical Optimizations:
- Model Choice: Used an ultra-small 28M parameter LLM, reaching 7 tokens/s for prompt processing and 4 tokens/s for generation.
- Architectural Tweaks: Abandoned cold-starting llama-cli for every interaction. Instead, a persistent llama-server process is used, leveraging cacheprompt=true to reuse the KV cache for stable prefixes.
- Performance Leap: By keeping the server resident and caching prompts, warm query latency for simple commands dropped drastically from 17 seconds to 2.3 seconds.
- Offline Voice Pipeline: Integrated Sherpa-ONNX for offline streaming STT. The LLM's output is strictly constrained to simple JSON action commands (e.g., controlling volume, toggling lights), bypassing the limitations of a tiny model to enable purely local IoT voice control.
More from Infra
- SK Hynix 2Q26 OPe Forecast Misses Consensus Due to High HBM Mix and LTA Effects — zephyr_z9 · 2026-08-07
- Building a Private RAG System for 50 Users: A Mac Mini Cluster Proposal — rogo725 · 2026-08-07
- Google TPU v7s Are Not Sold Cheap, Hardware Costs Remain High — zephyr_z9 · 2026-08-07
- $2900 for 64GB VRAM? Dev Weighs AMD GPU Upgrade Headaches — milkipedia · 2026-08-07
- Indie Developers Report Meta AI Scrapers Overloading Their Servers — Polymarket · 2026-08-07
- Report: DeepMind Gets Only 15% of GCP's Total Compute Resources — zephyr_z9 · 2026-08-07