DeepSeek V4-Flash Costs 105x Less, But Stability and Benchmark Overfitting Questioned

DeepSeek V4-Flash-0731 has recently demonstrated exceptional cost-effectiveness on multiple benchmarks, with Artificial Analysis reporting that its overall cost to complete the same tasks is 105x lower than competitor Fable 5. This has prompted calls to shift focus from traditional per-token pricing to per-task cost. Despite impressive scores, the model's API stability, overall capability, and billing fairness remain disputed, and it is highly sensitive to test frameworks.

Confirmed

• In Terminal-Bench 2.1, DeepSeek V4-Flash scored 82.7, nearly matching Fable 5's 80.5.
• Artificial Analysis reports DeepSeek's overall cost for the same benchmark tasks is 105x lower than Fable 5, confirmed by @zainhas and @mustafamhus.
• Output pricing is $0.28/M, exactly in the same range as OpenAI's GPT-5.6 Luna after an 80% price cut (plus OpenRouter promo to $0.60/M).
• LM Studio has listed the model, supporting local or cloud use, and it shows cost-effective advantages in codebase learning and modification tasks.
• The model offers Low, High, and Max effort levels (no Medium). @teortaxesTex notes DeepSeek needs Max to match Grok 4.5's M-level performance.

Unconfirmed

• API stability and actual total cost: Developer @kmsdev tested 34 prompts and found unstable provider performance on OpenRouter, multiple generation failures, and a final cost of $1.29, losing to Kimi K3 in both performance and cost.
• Misleading per-token pricing: @cephaloform and investor Chamath point out that despite low per-token prices, if the model needs more rounds to complete tasks, actual total cost may be higher.
• Extreme sensitivity to test frameworks: @PMinervini and @teortaxesTex found DeepSeek V4 Flash is very sensitive to test frameworks. With mismatched toolchains (e.g., Pi, Claude Code), performance degrades or even burns millions of tokens; with the right framework, it shows near-GPT-level cost-effectiveness.
• Benchmark overfitting controversy: @rickasaurus argues current LLM benchmarks are severely overfitted, noting that 5 months ago GPT-4o scored 51 on Artificial Analysis Intelligence Index, and this week DeepSeek V4-Flash also scored 51, rendering the charts meaningless.

Why it matters

• Shift in evaluation metrics: @mustafamhus and @AravSrinivas note that a two-order-of-magnitude cost reduction is extremely rare, calling it DeepSeek's "2.0 moment." This requires decision-makers and developers to move beyond per-token pricing and focus on "per-task cost." @intellectronica calls it a moment when AI is too cheap to meter. @svenai adds that DeepSeek achieves 50 intelligence points at a cost of 3 cents, highlighting extreme value.
• Open-source model comparison: In specific game generation tests (@mustafamhus), DeepSeek V4-Flash, despite low cost, still trails Kimi K3 (9.5/10, $0.0740) and GLM 5.2 (9/10, $0.0480) in scores, showing that cost reduction alone cannot replace overall capability competition.

2026-08-01 ~ 2026-08-03 · 17 related posts

Full story(15 episodes)→

• Episode 1: DeepSeek-V4-Flash Architecture Leaked with Million-Token Context(2026-07-31, 3 posts)
• Episode 2: Unsloth Releases Quantized DeepSeek V4 Flash 0731 for Local Deployment(2026-07-31, 6 posts)
• Episode 3: DeepSeek on Huawei Ascend Beats OpenAI in Inference Profitability(2026-08-01, 2 posts)
• Episode 4: DeepSeek-V4-Flash Excels in Frontend Coding with Unmatched Cost-Performance(2026-08-01, 3 posts)
• Episode 5: DeepSeek V4 Preview: Flash to Introduce Four-Level Reasoning Effort(2026-08-01, 2 posts)
• Episode 6: DeepSeek Drastically Reduces Training Compute Costs Across Models(2026-08-01, 2 posts)
• Episode 7: DeepSeek V4-Flash API Public Beta Launches with Major Agent Upgrades(2026-08-01, 5 posts)
• Episode 8: DeepSeek V4-Flash Costs 105x Less, But Stability and Benchmark Overfitting Questioned(2026-08-01, 17 posts)
• Episode 9: DeepSeek V4 Flash Sparks a Wave of Local Deployment Tests(2026-08-01, 26 posts)
• Episode 10: DeepSeek V4 Flash Quantization Tests Show Major Speed Gains Without Quality Loss(2026-08-04, 2 posts)
• Episode 11: Single RTX 5090 Runs DeepSeek 1M Context(2026-08-04, 2 posts)
• Episode 12: DeepSeek-V4-Flash Tops Cost-Efficiency with Ultra-Low Running Costs(2026-08-05, 4 posts)
• Episode 13: DeepSeek V4 Flash Benchmarks Leak, Outperforming Pro(2026-08-05, 3 posts)
• Episode 14: DeepSeek V4 Flash Tops ARC-AGI Cost-Performance, Costing a Quarter of GPT-5.6(2026-08-08, 9 posts)
• Episode 15: DeepSeek V4 Flash Passes 22 Coding Tests on Dual DGX Spark Cluster(2026-08-10, 2 posts)

• [source] DeepSeek-V4-Flash Eval: Underperforms Kimi K3 in Both Cost and Score — kms_dev · 2026-08-01
• DeepSeek V4-Flash completes benchmarks at 105x lower cost, but per-token price may mislead — cephaloform · 2026-08-02
• DeepSeek V4-Flash Reportedly Achieves Same Benchmark at 105x Lower Cost — AravSrinivas · 2026-08-02
• DeepSeek Completes Same Benchmark Tasks at 105x Lower Cost Than Competitors — JosephJacks_ · 2026-08-02
• [source] DeepSeek V4-Flash Completes Benchmarks at 1/105th the Cost of Fable — zainhas · 2026-08-02
• [source] DeepSeek V4-Flash Rivals Fable 5: The AI War Shifts to Intelligence-Per-Dollar — mustafamhus · 2026-08-02
• Kimi K3 vs GLM 5.2: DeepSeek V4 Flash is 150x Cheaper — mustafamhus · 2026-08-02
• DeepSeek V4 Flash Launch: AI Inference Costs 'Too Cheap to Meter' — intellectronica · 2026-08-02
• DeepSeek V4 Flash vs. OpenAI Luna: A Same-Tier Price and Performance Showdown — eyishazyer · 2026-08-02
• DeepSeek V4 Flash Performance Varies Wildly Across Coding Agents — PMinervini · 2026-08-03
• DeepSeek V4 Flash Matches Competitors in Coding at 1/34th the Cost — mattturck · 2026-08-03
• DeepSeek V4-Flash Nears GPT-4o Score, Community Calls Benchmarks Overfit — rickasaurus · 2026-08-03
• DeepSeek Dubbed 'Price Butcher': 3 Cents for 50-Point Intelligence — sven_ai · 2026-08-03
• DeepSeek V4-Flash Review: Needs Max Effort to Tie Grok, but Cost Per Task is Insanely Low — teortaxesTex · 2026-08-03
• DeepSeek V4 Flash Tested: Choosing the Wrong Harness Burns Millions of Tokens — teortaxesTex · 2026-08-03
• DeepSeek-V4-Flash Leaks: $2 Input / $6 Output, Runs on Single RTX 3090 — Boring_Aioli7916 · 2026-08-03

1 near-duplicate retellings: mustafamhus