Open-Source hunch Distills LLM Judgments Onto CPU: Millisecond Decisions Under $0.03/Million Tokens
aigclink · x · 2026-11-05
Open-source project hunch (347 GitHub stars) builds a locally-learning semantic decision system on top of Google's EmbeddingGemma 2:
- Performance: 22ms for text, 100ms for image judgments, running on CPU, under 3 cents per million tokens
- Architecture: EmbeddingGemma 2 + lightweight classifier + teacher-model distillation. A frontier LLM acts as the teacher; the local small model learns on the job and gradually takes over high-frequency, stable judgments, escalating hard cases back to the LLM
- Use cases: checking whether tool calls failed, spotting anomalies in logs, detecting prompt injections, user-intent classification, and routing tasks among agents
It's a "System One" fast-thinking layer that forms a continuously-learning, cost-shrinking loop for high-frequency AI decisions.
Related event: Open-Source hunch Delivers Millisecond Local Decisions on EmbeddingGemma 2(3 posts)→
More from coding & agent
- Anthropic shares Claude Code trick: run Sonnet, Haiku and Opus as builder, scout and advisor — Arindam_1729 · 2026-11-05
- Laya 0.4.24 lets an 0.8B model learn your labels without fine-tuning, with 73k-token context — NirantK · 2026-11-05
- Running a Hermes agent on uncensored Qwen3.6-35B-A3B, config shared — EvilEnginer · 2026-11-05
- Ghosthand gives AI agents a real iPhone to beat captchas and bot detection — ethanniser · 2026-11-05
- Biomedical benchmark: prewritten Cypher MCP tools hit 98% vs 85% for Text2Cypher — JeremyCMorgan · 2026-11-05
- Wise's answer to agent payments: agree once, then the agent acts within terms — AI Engineer · 2026-11-05