Towards AI Built a Benchmark for Detecting AI Slop Writing: 10 Scripts, 4 LLM Judges, 1000 Human Checks
lmoroney · x · 2026-10-23
Louis-François Bouchard of Towards AI presented his AI Engineer talk on a benchmark he spent 5 months building and released publicly, aimed at quantifying why AI writing feels like slop.
- Problem: overused words, vague generalities, saying nothing, repetition, hallucination, and a voice that sounds like everyone at once.
- Method: 10 video scripts, 5 drafts per model, scored by 4 LLM judges across 9 metrics, validated with 1,000 human A/B checks.
- Findings: judges favor their own model family; he also disclosed what the evaluation cost.
- Practical tips: very long prompts packed with examples for writing in a personal voice.
A rare first-hand, complete writeup on evaluating AI writing quality and benchmark design.
More from Research
- FreshStack retrieval leaderboard revamped, adds Cohere Embed v5 and EmbeddingGemma2 — beirmug · 2026-10-23
- Jakob Kather at ESMO: LLMs and AI agents in oncology, from single-task models to autonomy — jnkath · 2026-10-23
- Quantizing a 9B model: 70% of damage sits in the first 8 layers, and perplexity hides the tail — DinnerPrestigious953 · 2026-10-23
- GenReasoning team reflects on six-month grind: lean startup, zero paid promotion — rosstaylor90 · 2026-10-23
- 33K-Star GitHub Repo ML-From-Scratch Rebuilds ML Algorithms in Pure Python — techNmak · 2026-10-23
- Dev ports Anthropic's RISC-V formal verification model from Rocq to Lean — xennygrimmato_ · 2026-10-23