Benchmark: Tool Calling Performance of Qwen 35B-A3B Variants
OsmanthusBloom · reddit · 2026-08-26
The author benchmarked the tool-calling capabilities of Qwen3.6-35B-A3B and its fine-tunes. Using the tool-eval-bench 2.6.0 suite (88 tests in Hardmode) on a cluster of 32GB V100s, the results showed that Ornith 1.5 and Tiel-Coder (based on Ornith) were the top performers, scoring close to Qwen3.8-27B and significantly higher than Qwen3.6-27B. KAT Coder also slightly outperformed the original 35B-A3B. The tests utilized llama.cpp with Q4 quants across multiple runs, accounting for context pressure.
More from Models
- Unredacted Anthropic filing exposes internal data-quality debate months before Opus 3 cutoff — zacharynado · 2026-10-15
- Google says teams across the company are using Gemini 4 Argon on its hardest engineering problems — CurieuxExplorer · 2026-10-15
- Perplexity's new embedding models: index with a 9B model, search with 0.6B — antoine_chaffin · 2026-10-15
- 797GB of n-gram counts in CPU RAM let 1B models match 57% more compute — PangWeiKoh · 2026-10-15
- Leak: Moonshot's Kimi flagship 'x4' done with pre-training, K3.1 in internal testing — lyraxana · 2026-10-15
- Unsloth ships GGUF quants for Nemotron 3.5: Bolt-74B runs in 24GB RAM — danielhanchen · 2026-10-15