Authorship attribution and aesthetic evaluation of AI poetry: a case study with Haiku
Livia Oddi, Simone Scardapane, Toru Sugimoto, Donatella Genovese
cs.CL, cs.AI
2026-09-14
Six models wrote Japanese 5-7-5 haiku; 144 Tokyo students judged them. GPT-5 accuracy was 0.493. Aesthetic scores predicted 'human-like' (AUC 0.793), not true authorship (0.505).
Most work on AI poetry either scores how good a poem feels, or tries to make generation better. Hitsuwari et al. (2023) already showed that people cannot reliably tell human haiku from AI ones, and that human-in-the-loop pieces scored highest on beauty. What was missing was a cross-model comparison that asks a sharper question: do the same cues that make a poem feel human also help readers name the true author?
This study, from Sapienza University of Rome and Shibaura Institute of Technology, locks the testbed to regular Japanese haiku: 5-7-5 mora, at least one seasonal word (kigo). The form is tight enough that six models can be compared under the same rules, and the readers are Japanese university students, not English-language outsiders scoring translated verse.
Fine-tuning was tried and dropped. Optuna plus LoRA drove training loss down, but the models memorized 5-7-5 templates instead of composing. Closed models cannot be fine-tuned in the same setup anyway. Generation therefore used Japanese few-shot prompts: six annotated examples per prompt, three lines, 5-7-5 mora, at least one kigo, no Latin letters or digits.
Identity wording mattered. "You are a Japanese-language haiku poet" beat the more idiomatic "You are a Japanese haiku poet." Mean perplexity on StableLM-7B fell from about 52 to 34; on Gemma-2B from about 6975 to 530. The language-framed line likely matches instruction data better. Decoding was line-by-line sampling (temperature 0.7, top-p 0.90), not beam search, which copied n-grams from the few-shot examples.
The six systems:
Each model contributed 12 haiku (three per season), 72 AI poems in total. The human set was 12 prize-winning contest poems. Constraint pass rates split hard: LLM-JP and Gemini 2.5 at 100%, GPT-5 at 91.7%, StableLM-7B at 62.5%.
144 students at Japanese universities in Tokyo rated the poems. Twelve questionnaires, twelve poems each (six AI, six human, shuffled), twelve raters per set. Seven 1-5 scales: fluency, haiku-like wording, poeticness, coherence, understandability, favourability, unexpectedness, plus a source guess and a confidence rating. Analysis used L2-regularized logistic regression and Bayesian binomial mixed models with crossed random intercepts for raters and items.
Recognition accuracy by generator:
| Model | Accuracy | Reading |
| GPT-5 | 0.493 | chance |
| Gemini 2.5 | 0.503 | chance |
| StableLM-7B | 0.524 | chance |
| LLM-JP | 0.590 | moderate |
| Gemma-2B | 0.611 | moderate |
| LLaMA-2 | 0.670 | most detectable |
GPT-5's confusion matrix is almost symmetric: 75 human poems called human, 69 called AI; 77 AI poems called human, 67 called AI. LLaMA-2 was easier on both sides (100 human correct, 93 AI correct). Across 84 unique poems, item-level accuracy spread widely, including below-chance items. Detectability tracks the poem more than a stable model fingerprint.
Mean impression scores sat in the 3-4 band. Fluency, haiku-like wording, and coherence were highest; unexpectedness sat near the midpoint. Human and AI means were close. Confidence often lagged the quality scores: the poem looked well-formed, the source still felt uncertain.
Participant background barely predicted recognition. Full-model cross-validated AUC was 0.416; language-only 0.456; no-language 0.432. Non-native Japanese speakers did worse. STEM students slightly outperformed humanities, which the paper flags as exploratory; AI-tool familiarity was not measured.
The split that matters is correct recognition versus perceived authorship. Impression ratings predicting whether the source guess was right reached AUC 0.505. Higher favourability slightly lowered the odds of being right (OR 0.83). The same ratings predicting whether a poem was judged human reached AUC 0.793, driven by favourability (OR 1.72), haiku-like wording (OR 1.36), and theme relevance (OR 1.33). In the appendix, Grok labeled every GPT-5, Gemini 2.5, and StableLM-7B sample as human, for 50% accuracy on a balanced set.
Aesthetic cues run an attribution heuristic. They do not diagnose the source.
For anyone scoring creative generation with humans, this is a measurement warning. Turing-style indistinguishability is not evidence of creativity, and the paper does not treat it as such. The sharper problem is that beauty, haiku-likeness, and theme fit systematically push a poem toward "this was written by a person," even when the generator is GPT-5. Rank models by aesthetic questionnaires and you may be ranking attribution bias.
There is a product implication too. On a tightly constrained short form, frontier closed models and some Japanese open models already sit at chance against native-speaking students. Content moderation that assumes readers will simply see the AI taste does not hold for haiku. Smaller open models are still catchable: ELYZA's Llama-2 7B was recognized 67% of the time, and StableLM-7B discarded 37.5% of candidates on form. Passing 5-7-5 is not the same as passing a native reader.
This is incremental evidence, not a new task. Hitsuwari 2023 already reported that people cannot tell. The contribution is pinning the dissociation onto six current models and a reproducible few-shot pipeline: aesthetics predict "feels human" at AUC 0.793 and true authorship at 0.505.
The authors list the main ones. The setting is few-shot regular Japanese haiku; do not export it to free verse or other arts. The 144 raters are university students in Tokyo, not the Japanese public. The human reference is prize-winning contest work, so the AI poems are being mixed with a high-end subset, not average amateur haiku. Six models, one prompt family, one sampling recipe; other decoding or later versions will move the numbers. The statistics are exploratory, several AUCs hug chance, and the sample is modest.
Two further caveats. Every questionnaire was 6 AI versus 6 human, and participants knew both sources were present, so the base rate was pinned at 50%. Real reading is not a balanced two-class test. Also, StableLM-7B passed form constraints only 62.5% of the time, yet its accepted poems sat at chance like GPT-5. The human study evaluates filtered successes, not the model's average draw. Grok-as-judge is an appendix probe with 12 poems per model; treating it as a detector score would overread the evidence.