EQ-Bench creative writing v3 update: author favors Muse Spark, warns of LLM-judge blind spots
On Sep 7, EQ-Bench author sampaech ran a subjective creative-writing comparison across GPT-6-Astra, Fable 5.1, Muse Spark 1.3 and Gemini 3.8 Flash, and updated the EQ-Bench Creative Writing v3 leaderboard. He then addressed whether creative-writing benchmarks can still be trusted, offering a methodological warning.
Confirmed
- His personal favorite was Muse Spark 1.3—not leaderboard-toppers GPT-6-Astra or Fable 5.1—because it follows instructions directly and lacks a strong "model voice."
- On the others: Fable 5.1 has settled into fixed patterns, criticized as having a Claude-like voice; he described GPT-6-Astra as unable to write decent paragraphs.
- The results were published as the EQ-Bench Creative Writing v3 leaderboard with measured data.
- He stressed that ranking first does not mean real humans prefer it; all LLM-judged writing evaluations should be discounted, and readers should judge samples themselves.
Why it matters
- The core issue: models optimized to please reward-model preferences suffer readability degradation that LLM judges are insensitive to, and the problem is worsening. This caveat is key to interpreting this and similar creative-writing benchmarks, and explains why LLM-scored rankings diverged from his own taste.
Timeline
- Sep 7: comparison published and v3 leaderboard updated; same-day response on benchmark trustworthiness highlighting systemic LLM-judge blind spots.
- Sep 8: discussion spread, with "#1 on the board ≠ human favorite" further relayed.
2026-09-07 ~ 2026-09-08 · 6 related posts
Primary sources
- Creative writing model bake-off: author prefers Muse Spark, says GPT-6 can't write paragraphs — sam_paech ·
- LLM judges are blind to reward-model-optimized readability decay in writing evals — sam_paech ·
- EQ-Bench Creative Writing v3 Updated: Muse Spark 1.3 Beats GPT-6-Astra and Fable on Style — sam_paech ·
- [source] Creative writing model bake-off: author prefers Muse Spark, says GPT-6 can't write paragraphs — sam_paech · 2026-09-07
- [source] EQ-Bench Creative Writing v3 Updated: Muse Spark 1.3 Beats GPT-6-Astra and Fable on Style — sam_paech · 2026-09-07
- [source] LLM judges are blind to reward-model-optimized readability decay in writing evals — sam_paech · 2026-09-07
- LLM-judged writing evals are blind to readability issues from reward model optimization — koltregaskes · 2026-09-07
- Creative writing leaderboard No.1 isn't what humans prefer, tester finds — FinanceYF5 · 2026-09-08
1 near-duplicate retellings: koltregaskes