Blind listening test of 62 raters ranks Gemini first, Llama last among 8 LLM composers

antiochIst · reddit · 2026-11-06

A self-run blind listening study gave 8 flagship LLMs one shot each at writing 8-bar solo piano loops (JSON note lists) from the same 20 prompts, rendered through the same sampled piano, with 62 paid lay listeners picking winners pairwise.

Interim results (514/1,638 judgments)

Gemini 4 Argon leads at 66% win rate (ranked first in 79% of bootstrap replicates), Claude Opus 5.5 second at 61%. The middle five (Qwen 3.8 Max, Kimi K3, Grok 4.7, DeepSeek V4 Pro, GPT-6.1 Terra Pro) can't be ordered yet. Llama 4 Maverick is last at 27%, dead last in 100% of replicates with zero wins on 4 of 20 prompts.
Preferred loops had wider pitch range (62%), more notes (59%), more bass (58%). Style effects are notable: Qwen hits 72% on dark prompts but 42% on uplifting; Grok only 32% on uplifting.
Losing composers write sparser music: DeepSeek 71 notes over two octaves; Llama 52 notes over 17 semitones with almost no bass register.

Limitations: 31% complete, one generation per prompt, lay listeners, Llama served via third-party FP8 endpoint.

Original post →

Blind listening test of 62 raters ranks Gemini first, Llama last among 8 LLM composers

More from Models