Deep Dive: How Prompts, Params, and Engines Skew LLM Benchmarks
rsasaki0109 · x · 2026-08-21
This blog analyzes common pitfalls in LLM evaluation. Scores are sensitive to:
- Prompts: Minor rewording or format instructions in zero-shot setups cause variance; beware of model-specific biases.
- maxnewtokens: Reasoning models need long chains; small limits truncate thoughts, leading to false negatives. Monitor truncation rates.
- Inference Params: Follow official recs (e.g., Qwen3.5 advises against greedy). Non-zero temperature requires multiple runs for mean/std.
- Precision & Engines: FP precision, inference engines (vLLM/SGLang), and API providers (e.g., OpenRouter) alter outputs.
Always verify if settings unlock the model's true capability before comparing.
More from Models
- Stealth Model 'Ox Alpha' Hits 80% on DeepSWE Benchmark — AccBalanced · 2026-08-21
- GLM-5.3 coding performance vs Fable 5 breakdown is respectable — zainhas · 2026-08-21
- GLM-5.3 beats Fable 5 on DeepSWE at 1/5th the cost: $3.99 vs $21.63 per task — zainhas · 2026-08-21
- Users report OpenAI Codex quota cuts and quality degradation — mazzaTalk · 2026-08-21
- Unsloth V3 Qwen Model Broken on Dual AMD GPUs, V2 Fix Available — Equivalent-Ear-8016 · 2026-08-21
- Rumor: Ox Alpha is a joint ZAI and GLM model — scaling01 · 2026-08-21