SimpleHarn Routes Local LLMs and Refuses Loads That Exceed Your VRAM

legislat · reddit · 2026-11-11

The author released SimpleHarn, a local model router/switcher that runs as a Windows tray app or headless Linux daemon and exposes an OpenAI-compatible API.

Core mechanism

It reads live VRAM via NVML (with an nvidia-smi fallback) and checks every model load against the card's reported budget. If a load doesn't fit, it refuses, naming the blockers and actual memory figures in the API response, control page and log — it never force-loads onto a full GPU.
Four category endpoints expose OpenAI-style APIs: WORK (8708, top priority), RP (8709), AUX (8710, multiple slots) and EMB (8711, lowest).
Eviction follows a fixed deterministic matrix: WORK is never evicted and only WORK evicts RP; WORK and RP can evict AUX and EMB; when several evictions would cover the deficit it picks the cheapest sufficient set. All load/unload decisions serialize through a single scheduler so client, tray and API can't race.

Other details

Default engine is llama.cpp's llama-server; Strata, Basalt, Maya, NInfer and Xolotl are also wired in.
Local-first: the API binds to 127.0.0.1 by default, and a non-loopback bind without a token is refused at boot (set SIMPLEHARNTOKEN).
Apache-2.0, Windows 10/11 and Linux, one NVIDIA GPU, single user. The author notes the project was built almost entirely by coding agents overnight, with design and architecture directed by hand; AGENTS.md documents the workflow.
The author is seeking testers, especially 8–16 GB card owners to test tight-budget refusals and Linux users to report on the fresh port.

Original post →

SimpleHarn Routes Local LLMs and Refuses Loads That Exceed Your VRAM

More from coding & agent