Specification Before Generation: A Pre-Registered, Five-Model Paired Evaluation of a Specification Frame for LLM-Generated Code in Money, Time, Idempotency, and Access Tasks
Sandeep Dhuri
cs.SE, cs.CR
2026-09-20
Pre-registered paired trial: 50 backend tasks across five frontier models, each run bare and with a 267-word spec prefix. Named defects fell from 148 to 23; no model got worse.
The pass rate of LLM-generated code on security checks has barely moved in four years. Veracode's 2026 report covers more than 100 models across four testing snapshots and puts the aggregate at 56 percent; individual models score higher, but the aggregate does not improve with scale. The standard industry answer is repository instruction files (AGENTS.md, CLAUDE.md, cursor rules) that tell the model how the team works. The largest controlled study of those files, at 438 tasks, found no gain in task success, sometimes a drop, and a cost penalty above 20 percent.
The diagnosis here: instruction files teach how to write, while the prompt is missing a specification, a statement of what must be true of the result. In regulated backends that list is short and stable: money never in binary floating point, timestamps carry a zone, retryable handlers check an idempotency key, passwords only as slow salted hashes. Rules any senior engineer could write from memory. The claim under test is one sentence: prepend that list to the prompt, and defects in the named classes go down.
The artifact is a fixed 267-word preamble, the Specification Frame, with four sections (Role, Context, Task, Constraints), instantiated once before generation and byte-identical across all tasks. Constraints state rules, not procedures: monetary amounts in decimal.Decimal or integer cents; retryable handlers must require and check a caller-supplied idempotency key.
Several choices are stricter than the norm:
The checkers and the frame share an author, which is the obvious hazard. The defense: checkers read the syntax tree, not prose, so echoing the frame's language scores nothing; and Bandit, with zero coupling to the frame, points the same way for every model.
Δ is findings per task, bare minus frame. All five bootstrap confidence intervals exclude zero and all Holm-adjusted sign tests are significant:
| Model | Bare | Frame | Mean Δ | Holm p |
| claude-sonnet-5 | 40 | 6 | 0.68 | 4e-5 |
| deepseek-v4-pro | 22 | 2 | 0.40 | 4e-5 |
| gemini-3.1-pro-preview | 44 | 9 | 0.70 | 4e-5 |
| qwen3.8-27b | 30 | 12 | 0.56 | 1e-5 |
| gpt-5.6-sol | 12 | 4 | 0.16 | 0.021 |
Of 250 pairs, 150 tied and 100 differed; the frame arm won 95 and lost 5. No model got worse in any domain. Pooled (descriptive per the pre-registration), named defects fell from 148 to 23, an 84 percent reduction. Bandit logged 53 medium-or-high issues in the bare arm against 11 in the frame arm, same direction for every model. By domain: money 66 to 8, idempotency 32 to 2, datetime 31 to 5, auth 19 to 8.
Effect size tracked each model's unprompted weakness, monotonically; this was not hypothesized. The cleanest bare arm (gpt-5.6-sol, 12 findings) gained the least (0.16), the dirtiest (gemini, 44) gained the most (0.70). The frame is a floor. The two tasks whose text already carried a timezone constraint produced zero findings in every model and arm, which is what a specification account predicts. A seeded manual audit of 18 checker hits found all true positives. Total API cost: under $20.
For backend teams this is close to a free intervention: short, task-agnostic, version-controlled with the code. Vendors rotate models under fixed product names, and a repository-level specification is the only part of the generation context a team fully owns. The two largest effects, money arithmetic and idempotency, sit outside security scanners' rule sets; a team measuring generated code by scanner findings alone will not see these defects and will not see the frame's biggest benefit. Domain checkers of the kind used here are cheap to write.
In kind, this is incremental prompt-engineering evidence, not a new method. The action it validates is a single one: state the constraints.
The author's list is unusually candid. Checkers and frame share an author (mitigated by AST scoring, frozen fixtures, Bandit corroboration). The frame arm adds 267 words, so part of the effect could be extra deliberation rather than specification content; no length-matched neutral preamble or reason-first arm was run, so that confound stands open. Functional correctness was not scored; a post hoc compile check came out 247/250 bare versus 236/250 frame, and RealSec-bench has shown security prompting can cost functionality, so a trade-off cannot be ruled out. Sixteen outputs were truncated in transport, 14 of them frame-arm, which could only bias in the frame's favor; excluding those pairs leaves every per-model conclusion intact. Fifty tasks, one author, Python only, single-turn, one run per arm.
Two reservations from this reading: the study is solo and self-funded and validates the frame from the author's own book, with pre-registration and full publication as the hedge; and the four addenda changed the roster and task count before generation, all dated, but the final design is not the one first registered.