Testing Local LLMs for Knowledge Base Agents
o0genesis0o · reddit · 2026-07-17
This is a hands-on evaluation of locally deployed LLMs. The author tested an agent workflow featuring memory, skills, and extensions on an **4060 Ti 16GB** across scenarios including: - Obsidian knowledge base management - Personal productivity assistance - Reading Gmail, connecting to project management systems, and nightly memory compression The author compared the performance of several local and cloud models within this complex workflow: - **Gemma 4 12B QAT**: Fast prefill (~1000 tok/s in deep context). Handles most tasks well but can be lazy, sometimes even "faking" tool calls. - **Gemma 4 26B QAT**: Behaves similarly to the 12B but is slower. - **Qwen 3.6 35B A3B**: Overall performance closest to minimax-m3 with higher summarization quality, though still on the slower side. The author concludes that: - Such complex agent workflows are already quite functional on local models. - However, bottlenecks remain: high KV cache usage, mediocre speeds in deep context, and compatibility issues with speculative decoding in the current version. - The core value of local deployment is the ability to reliably support complex knowledge base or assistant workflows without relying on the cloud.
More from coding & agent
- Autoresearch proposes packaging ML runs as studies with questions, analysis, and code diffs — morgymcg · 2026-07-21
- CHAP defines approvals, handoffs, and audit logs for human-agent workflows — DeliveryTechnical199 · 2026-07-21
- The author says Codex reached 20x and is now debugging spec decoding on a hybrid parallel setup — TheZachMueller · 2026-07-21
- Axcess adds an MCP connector for WCAG accessibility checks that scanners miss — modelcontextprotocol · 2026-07-21
- X post asks whether Cursor Composer, built on Kimi models, would also be banned — max_paperclips · 2026-07-21
- A developer’s Codex usage is draining pooled enterprise credits at a small company — Distinct_Relation_62 · 2026-07-21