Hacker Project: Running Local LLMs on an Echo Dot 2 with Only 512MB RAM
alberto_zurini · reddit · 2026-08-07
A developer successfully deployed a fully offline voice assistant pipeline on an Amazon Echo Dot 2—equipped with only an ARMv7 processor and 512MB of RAM—achieving local speech recognition and LLM inference.
Core Technical Optimizations:
- Model Choice: Used an ultra-small 28M parameter LLM, reaching 7 tokens/s for prompt processing and 4 tokens/s for generation.
- Architectural Tweaks: Abandoned cold-starting llama-cli for every interaction. Instead, a persistent llama-server process is used, leveraging cacheprompt=true to reuse the KV cache for stable prefixes.
- Performance Leap: By keeping the server resident and caching prompts, warm query latency for simple commands dropped drastically from 17 seconds to 2.3 seconds.
- Offline Voice Pipeline: Integrated Sherpa-ONNX for offline streaming STT. The LLM's output is strictly constrained to simple JSON action commands (e.g., controlling volume, toggling lights), bypassing the limitations of a tiny model to enable purely local IoT voice control.
More from Infra
- Amazon Proposes Per-Request Validated Evidence Caching for Production RAG — _reachsumit · 2026-10-16
- LinkedIn's TransX report: sequence transduction for recsys, +6.02% CTR in A/B tests — _reachsumit · 2026-10-16
- llama.cpp PR boosts CPU inference speed with merged AVX2 tiled accumulators — jacek2023 · 2026-10-16
- Zero's next-gen IVM engine runs in a single DO, live-querying a 100GiB R2 database — threepointone · 2026-10-16
- Meme: driving to the chip factory in 2028 captures AI's compute anxiety — beffjezos · 2026-10-16
- One volunteer maintains the time zone file behind 4 billion phones — danbri · 2026-10-16