Running agent swarms under 6GB VRAM: millisecond micro-models as workers, LLM only for decisions
everettplays · reddit · 2026-10-18
A developer shares a decoupled agent-swarm architecture for constrained local hardware (<6GB VRAM), replacing the common pattern of 4-8 large model instances burning context on routine triage:
- Orchestrator: a single dense 30B-40B reasoning model (or API) handling multi-step decomposition, planning, and state updates; idle during routine operations
- Worker swarm: fleet of custom narrow nano-models (1M-3M params, dubbed "ants") on CPU threads doing reflex tasks — signal/log triage, regex extraction, anomaly scoring, schema coercion at sub-millisecond speeds
- Escalation protocol: ants escalate to the orchestrator only on low confidence or unhandled edge cases
The goal is an air-gapped, zero-cloud system where workers consume near-zero power/RAM. The author asks the community about queue backpressure, micro-tokenizers vs byte-level embeddings for sub-5M models, and pitfalls bridging sub-ms workers with slower reasoning brains.
More from coding & agent
- DeepSeek V4.1 Flash 1017 aces 10-task coding benchmark at under 5 cents per task — Fabulous_Pollution10 · 2026-10-18
- dotey: Rust compile times slow AI agents, so he's back to Electron except for core components — dotey · 2026-10-18
- 116 agents, 2.4B tokens: SI assistant 'passes the tests' by deleting them — tetsuoai · 2026-10-18
- 3D chess game with AI coach built entirely in Codex using GPT-6.1 Sol — DeryaTR_ · 2026-10-18
- Claude Opus 5.5 ports a web game to Godot and ships native mobile builds in hours — majidmanzarpour · 2026-10-18
- Speed up your repos: APFS volume with no fseventsd or Spotlight, via your agent — GabGarrett · 2026-10-18