What benchmarks are worth running locally? Reddit thread on DIY inference evals
JustTooKrul · reddit · 2026-09-17
A local inference user asked r/LocalLLaMA for advice on building a reproducible local eval suite: measuring quality loss from KV cache quantization (BF16→Q8), speculative decoding performance with different MTP/drafters, and tool calling across chat templates (referencing early Qwen 3 tool-calling pain).
They prefer deterministic benchmarks to avoid LLM-as-judge noise, plus long-context-capable tests. Currently implemented: MMLU-Pro, GPQA-Diamond, BCFL v4, RULER; planned: LiveCodeBench, Terminal-Bench, PlanBench. The concern is over-engineering—running benchmarks that only reflect overall model quality rather than real differences between vLLM/llama.cpp configs. The thread solicits what the community actually runs locally.
More from coding & agent
- From babysitting simple features to full monoliths by agents in 8 months — reach_vb · 2026-09-17
- AI Engineer Search Track: 30-Year-Old BM25 Emerges as Backbone of Agentic Search — jobergum · 2026-09-17
- Innate OS Open-Sourced: Intuitive Agentic OS for General-Purpose Robots Under $1K — chris_j_paxton · 2026-09-17
- Antigravity 0-day shows how DNS rebinding on loopback turns AI coding agents into RCE — evilsocket · 2026-09-17
- Astra designs, Luna executes: a quota-saving multi-agent coding workflow with scheduled reviews — TheMoonMidas · 2026-09-17
- Dev Open-Sources Blue to Fix MCP Delivery and Config Pain in Inference Gateways — Educational-Hold9425 · 2026-09-17