2-bit Bonsai-27B fits in 8GB VRAM but trails Qwen3.5-9B on Terminal-Bench
Creative-Regular6799 · reddit · 2026-07-21
A Reddit user benchmarked Ternary-Bonsai-27B at 2-bit and Bonsai-27B at 1-bit on Terminal-Bench 2.0 using an agent harness on an RTX 5070 Laptop with 8GB VRAM.
What they ran
- 89 tasks from Terminal-Bench 2.0
- Single-attempt runs with a 40-turn cap and temperature 0.2
- llama.cpp-based runtime via PrismML, because stock llama.cpp could not load the 2-bit kernels
Results
- Ternary-Bonsai-27B (2-bit): 7.9%
- Qwen3.5-9B: 9.2%
- Qwen3.6-35B-A3B: 24.3%
- Bonsai-27B (1-bit): not viable in the agent setup
The key takeaway is that the 2-bit model does fit entirely on the GPU and tool calling stayed clean, but accuracy was worse than the smaller dense Qwen baseline. The 1-bit model was usable on simple prompts, but in an agent loop it produced a runaway 14,000+ token completion and never emitted a stop token, eventually exhausting the context window.
More from coding & agent
- Codex helps build Valdiluce, an open-world game with climbing, gliding and gondolas — Dimillian · 2026-07-22
- HeyGen adds a media-sourcing skill for coding agents with 75k images and 10k tracks — HeyGen · 2026-07-22
- Agent search bottlenecks are now about variance, not raw latency — rohanpaul_ai · 2026-07-22
- LangSmith adds tracing for Pipecat, LiveKit, OpenAI Realtime, and Gemini Live — LangChain · 2026-07-22
- An MCP server signs every AI agent tool call into a verifiable Merkle chain — Funky_Chicken_22 · 2026-07-22
- Annotated transcript of a Claude Code team interview is now available — trq212 · 2026-07-22