Two weeks of service_tier "priority" A/B tests: median +5.5% TTFT, basically noise
iambubu0523 · reddit · 2026-11-18
A developer built a Fast Mode for DeepSeek Harness (DSH) that injects servicetier: "priority" via a fetch hook after JSON serialization, then honestly measured its effect on first-token latency with 8 interleaved A/B probes.
- Why not the obvious shortcuts: prompting for brevity, auto-lowering reasoning effort, or swapping to a smaller model all change model behavior; priority is the only mechanism keeping model, reasoning effort, and output length identical.
- Measurement design: standard/fast arms interleaved within the same minute, TTFT measured from the first text-delta on the stream, with per-request assertions that the parameter actually reached the wire.
- Result: the initial single-round +23.6% was noise; median over 8 probes was just +5.5%. The parameter was accepted (no 400/422) but the server did nothing — likely the norm for most relays.
- Three rules codified: <3 samples returns inconclusive; verdicts aggregate the last 5 probes per host|model and need ≥2 decisive wins of ≥15%; 400/422 is a hard never-retry signal.
- Installable via github:iambubu0523/dsh-turbo (MIT, zero deps, 46 tests), with a per-session Fast toggle and a turbomode agent tool; the author asks others to probe OpenAI's real priority tier and share numbers.
More from coding & agent
- Open-sourced genetic algorithm optimizer hit top 2 in Open Kernel Challenge within 7 hours — morgymcg · 2026-11-18
- Future codebases will be 80% AI-assisted, raising sloppier-code concerns — 996roma · 2026-11-18
- Sentry founder jokes about MCP as HN thread declares it the standard for AI agent tooling — zeeg · 2026-11-18
- Self-hosted Qwen3.8-27B + pi LoRA matches Opus 5.5 on a real bug fix in ~93s — yb2698 · 2026-11-18
- Astra 6.2 hailed as best computer-use model after flawless MS Paint demo — gabrielchua · 2026-11-18
- Driveline team's model lineup: Opus 5.5 still king for coding, Carbon 4 for reviews — ivan_bezdomny · 2026-11-18