Beating the Style Detector: Three Hours of Agentic Research on the AI-Text Arms Race
Andreas Maier, Moritz Zaiss, Siming Bayer
cs.CL, cs.LG
2026-05-04
An agentic harness reproduces all seven ACL 2026 style hypotheses in three hours. GPT-5.5 and Opus 4.7 close 71-75% of the style gap; 20 rewrites flip 2 of 5 detector calls.
Baumler et al. (ACL 2026) asked a narrow question: if people post-edit an o4-mini draft until it "sounds like them," how far does style actually move? 81 participants, 486 task observations, style scored as cosine similarity in LUAR-MUD, an authorship embedding trained on Reddit. Humans take a small step; a large residual remains. The 81 JSON logs shipped; the analysis code did not. This Track 2 companion rebuilds the stats pipeline with an agentic research harness and adds three experiments. The human is a reviewer-in-the-loop. Wall-clock time is about three hours, not weeks.
The live question is no longer whether human post-edits help. It is whether a frontier model, given one genuine sample and writing from scratch, can close the personal-style gap, and whether a detector still catches it afterwards.
The reproduction tidies the 81 logs into a long table, pins LUAR-MUD at revision 9204529, caches embeddings as .npz, and runs the rest on CPU. Group tests are two-sided paired permutations (10,000 shuffles, smallest reportable p about 10^{-4}), effects are small-sample Hedges' g, and the seven preregistered tests use BH-FDR at q=0.05. Perceived versus LUAR self-similarity uses repeated-measures correlation.
The new protocol splits each participant's two unassisted controls: the lower taskidx is the demo shown to the generator; the other is a held-out target nobody sees. The o4-mini draft, the human post-edit, a GPT-5.5 mimic, and a Claude Opus 4.7 mimic are all scored against that same vector. An earlier iteration showed both controls to the model and scored against their centroid, i.e. the answer sat in context. Under the strict protocol, Hedges' g for "LLM mimic versus human post-edit" shrank from +1.18 to +1.02. Only held-out v1 is reported.
Detection puts the same 512-d embeddings through a linear SVM with leave-authors-out 5-fold GroupKFold, so each author appears on exactly one side. The human class is 81 people × 2 controls (162); the AI class is that approach's 324 outputs. Six diagnostics check author leakage, shuffled labels, length-only, cross-model transfer, PCA(32), and strong L2. The adversarial loop freezes the fold-1 Opus detector trained on 64 authors, takes the five most AI-like Opus drafts among 17 held-out authors, and lets Opus 4.7 sub-agents see the signed margin for T=20 rewrites, keeping content and the 100–200 word band.
All seven preregistered hypotheses replicate in sign and survive BH-FDR. H3's rmcorr matches to three decimals: r=+0.244, p=3.6×10^{-9}, n=648, against the paper's +0.244±0.076. Four tests that average similarity to other participants shift in magnitude because the source never specified pairwise versus per-person first; per-person means recover the reported directions, and match H1a, H2a, and H2c in size.
On 324 paired tasks, mean held-out LUAR cosine is 0.498 (o4-mini), 0.546 (human post-edit), 0.643 (Opus), 0.649 (GPT-5.5). Same-author control-versus-control sits at 0.701. Relative to that ceiling, humans close about 24% of the gap; the two frontier models close 71–75%. Friedman χ²(3, n=324)=419.3, p=1.5×10^{-90}. Five of six pairs survive FDR; Opus versus GPT-5.5 is a tie (g=−0.08, p=0.14). Per-task win rate versus the human: o4-mini 18.5% (37/324 plus 46 ties), Opus 76.9% (249/324), GPT-5.5 79.0% (256/324). The order holds in all eight scenarios, from thank-you letters to wedding vows.
| Approach | held-out mean | win vs human | detection AUC |
| o4-mini draft | 0.498 | 18.5% | 0.999 |
| human post-edit | 0.546 | — | 0.971 |
| Opus 4.7 mimic | 0.643 | 76.9% | 0.952 |
| GPT-5.5 mimic | 0.649 | 79.0% | 0.931 |
AUC falls from 0.999 to 0.931, with non-overlapping bootstrap CIs between adjacent rows. Length alone reaches 0.880 for GPT-5.5 and 0.517 for Opus: GPT-5.5 detection is mostly length, Opus detection is style. Cross-model transfer still hits 0.913 / 0.888, so a shared frontier signature exists. In the rewrite loop, mean margin drops from +3.13 to −0.16; two of five drafts cross into the human half-space and stay, a third finishes 0.04 from the boundary, and none plateaus on the AI side.
An NLP study that released data but not code can now be rebuilt, leakage-checked, and extended in about three hours if an agentic harness is on hand. For AI-text detection, the snapshot is asymmetric: mean style is already close, yet the LUAR cloud still has enough shape for a linear SVM. What the SVM grabs differs by model. GPT-5.5 is mostly length; Opus is style. Show a frozen detector's score and give the model 20 rewrites, and two of five held-out drafts flip. Against a known detector, at moderate effort, a frontier LLM can already push down its own AI-detection probability. The 648 mimics, trained detectors, and trajectories are public.
Generation and scoring are model-on-model. LUAR recovered r=0.244 to three decimals, but a third independent generator is still missing. Humans edited an unconditioned o4-mini draft; the mimics saw a style sample and wrote from scratch. That compares workflows, not raw writing skill. LUAR scores resemblance to an author, not trust, factuality, or whether the person would sign the draft; the source paper's §6.3 already found perceived authenticity and LUAR can split. One demo per author, so demonstration-set size is untested. The linear SVM is a lower bound on detectability; 20 rewrite rounds are an upper bound on what this agent already removes. Absolute AUCs are a spring 2026 snapshot. One Opus draft overlaps its demo at 0.56 lexical similarity; the mean shifts by less than 0.001, disclosed and kept.