Melbourne plan: humans judge a 1,500-word brief, models verify the appendix

Ignore the Writing: A Proposal for Peer Review in the Age of AI

Vassilis Kostakos

cs.HC, cs.DL

2026-10-28

A Melbourne proposal limits human review to a 1,500-word brief on whether the work is worth doing, and has models re-run an unlimited appendix. It reports no experiment.

What problem this solves

A polished paper used to function as proof of work. Clear prose stood in for clear thinking, and a thorough description stood in for a thorough method. That proxy was tolerable only while good writing was expensive. Generative models removed the cost. Length is nearly free, competent academic prose can be produced on demand, and text quality flattens across submissions. How much was written, and how well, no longer sorts good work from bad.

Submissions already outrun what reviewers can read. Vassilis Kostakos, at the University of Melbourne, names the endpoint directly: models write the papers, and models review them. Text production is faster than human reading, so the rules for that equilibrium have to be set now.

Each tradition expects automation to hit the other one. Qualitative researchers expect software to absorb quantitative work. Quantitative researchers expect a model to turn interview transcripts into themes. What can be automated is the mechanical middle of both. The ends remain: which questions are worth studying, and whether a solution is actually desirable in the world. The long human-written paper is the one object peer review knows how to judge, and it is the object disappearing.

Method

A submission becomes a 1,500-word brief plus an appendix with no length cap. The brief is the only human-facing text. It states the idea and why the idea has value. The appendix carries procedures, results, analyses, data, related work, discussion, and references.

Machine learning conferences have run short page limits with unlimited appendices for years, and the reviewing crisis did not ease. Changing the length of the text, without changing what humans do with it, does not fix the bottleneck. The proposal changes the criteria and the division of labor.

Humans read the brief, and only the brief counts. Three questions: is the pursuit worth doing, is the approach convincing, and are the likely gains worth the effort? Research funders have asked this kind of blunt question for decades. The paper treats it as the judgment models are worst at.

Models verify the appendix. They recompute the statistics, re-execute the code, and re-query the data, then place a checklist next to the brief. Both gates must pass. The reviewer sees the brief and the checklist, not a methods section taken on faith. Acceptance is redefined as curation: the venue says the work deserves the field's attention, and the full record stays archived. Nature and Science are cited only as places where editors already judge importance from a short main text. The paper does not claim those journals have handed verification to models.

Models can read, extend, and re-run the literature faster than a human pipeline. That speed is presented as the loop's one real advantage, and it is conditional on a reward function. The term is borrowed from reinforcement learning, where it means the score that decides which behavior is repeated. Human reviewing effort is supposed to be that score, choosing where finite compute goes. The space of ideas has no bound, and every generated paper and every verification run spends finite energy.

Results

There is no experiment. No timings, agreement rates, or miss rates, and no quantitative comparison with reviewing as practiced now. The paper calls itself a provocation, leaves the design under-specified on purpose, and cites nothing. The document itself is under 1,500 words, and portions were drafted with a language model.

PieceWhat the paper says happens nowProposal
What humans readThe full manuscript; length and prose act as proof of workOnly the 1,500-word brief
What humans judgeQuality, conduct, and communicationWorth doing; whether the approach is convincing; whether gains cover the effort
Experimental recordHumans skim the methodsModels recompute, re-execute, and re-query, then emit a checklist
What acceptance meansA stamp of approvalCuration: this deserves to continue
Length precedentML venues already use a short main text plus an unlimited appendixThe short part is the human task

The paper gives no direct comparison on reliability or cost. The only quantities it states are the 1,500-word cap, the three questions, and the claim that reviewers are probably already feeding twenty-page papers to a model. A weekend of generation that exceeds what a subcommittee can read in a cycle is stated with no count of papers.

The objections stay argumentative. The worry that methods will be skipped is answered by saying a re-run is a stronger check than a tired reviewer skimming on faith, and that a brief whose claims cannot be traced into the record fails. Reviewers feeding the brief to a model are described as already normal for twenty-page papers. The paper grants that worthwhile is more subjective than the criteria it replaces, and that a model could choose the field's goals about as rationally as humans. Taste already dresses itself up as soundness, so making the value judgment explicit is what makes it contestable. Choosing goals is about values held by the people who live with the outcomes. An essay can sit whole in the appendix. On homogenisation, the claim is that today's house style polices every page, while a fixed brief shrinks the uniform surface and takes the appendix out of style grading.

Why it matters

Shortening the manuscript without changing what humans evaluate leaves the reviewing load where it is. ML conferences are the precedent offered for that failure mode.

If human time moves from polishing prose and trusting a methods section, onto whether a direction deserves more of the community's compute, reviewing occupies the slot a reward function has to fill. Authors then face a concrete gate: the case has to survive three questions, and an experimental record has to be re-runnable. Essay-style work is filed whole in the appendix, outside that check.

The role split is structural. The evidence for it is empty. It can frame a debate. It is not a venue policy ready to operate.

Limitations

The paper states its own open questions. How many reviewers judge a brief, how worthwhile is calibrated across people and venues, and who audits the verifiers are all left for later tests. Citing nothing, and drafting portions with a model, is paired with an instruction to ignore the prose and judge the pursuit.

Verification is written as if models can already recompute, re-execute, and re-query. There is no design for irreproducible experiments, closed weights, private data, human evaluation, or hardware-tied results, and no rule for who decides when the checklist contradicts the brief. The reward function stays a metaphor. The signal could be acceptance, a score, citations, or a compute budget, and none is specified. How that signal would feed the models writing the next papers is undefined.

Making the value judgment explicit is not shown, by any review experiment, to make it more contestable. Explicit taste can also concentrate among a few senior reviewers and editors. Nature and Science still have humans reading the supplement, so the analogy does not reach an appendix that nobody reads. The claims that the ML reviewing crisis is worse, and that a weekend outproduces a subcommittee, come with no numbers. Because nothing is cited, funders have done this for decades and reviewers already feed twenty-page papers to models cannot be checked either.

Terms

Source

What people are saying

Related papers

All paper explainers