Help needed: Native 64k+ context GGUF model for llama.cpp
Inner-End7733 · reddit · 2026-07-31
A developer encountered a context bottleneck while trying to fully localize Hermes Agent on low-spec hardware (64GB RAM + RTX 3060).
The Problem:
- Running a quantized Qwen 3.5 35B primarily in CPU mode at an acceptable speed (30 t/s).
- Hit a wall configuring a secondary model for context compaction: Hermes Agent hardcodes a 64k context request.
- If the GGUF model's native training context (nctxtrain) is smaller than 64k, llama.cpp rejects the task outright, regardless of RoPE scaling or flag settings.
The Request:
Looking for a dense, non-reasoning model natively supporting 64k+ context, already quantized in GGUF, to meet the strict requirements of the local agent workflow.
More from coding & agent
- shadcn registry becoming the standard package manager for AI agents — shadcn · 2026-07-31
- Airbnb Tries Eval-Driven Development for Building AI Systems — rseroter · 2026-07-31
- Open-Sourcing Watchword: A Cross-Agent Memory KV Store Built on MCP — glani-ccw · 2026-07-31
- AI Coding Tool Introduces 'Computer Brain' for Project Memory Self-Improvement — AravSrinivas · 2026-07-31
- Opinion: Use LLMs as an Intelligence Layer, Not Just Text Output — braelyn_ai · 2026-07-31
- Viral Open-Source Tool Uses LLMs for Deep Recursive Auto-Research — natesiggard · 2026-07-31