LLM fair-use eval leak: Lawrence Lessig answer scores 9, rest cluster at 5-8

dejanseo · x · 2026-11-13

In a reply to a Karpathy thread, dejanseo shared an LLM evaluation on the task of assessing whether training AI on copyrighted books without a license counts as fair use. An answer attributed to law scholar Lawrence Lessig scored 9, the only one correctly covering the 2025 rulings (Bartz v. Anthropic, Kadrey v. Meta, Thomson Reuters v. Ross); the other expert and anonymous answers clustered at 5-8. Scoring factored in case-law currency, presence of a clear verdict, and answer/thinking token counts.

Original post →

LLM fair-use eval leak: Lawrence Lessig answer scores 9, rest cluster at 5-8

More from Models