2026-09-24
Frozen A/B: gpt-4-0613 gave zero-byte normal-stop replies on 30/30 null prompts; gpt-3.5-turbo-1106 leaked visible text every time. Both spoke on all 30 controls.
Early GPT-4 studies asked what the model could do. Microsoft Research's Sparks of AGI paper measured breadth across tasks. OpenAI's launch note talked about factuality, refusals, and six months of alignment work. This paper asks a narrower, checkable question: given identical prompts and API settings, does the served system emit visible text at all.
Rayan Pal, an independent researcher, records one kind of successful blank as V0: HTTP 200, a present text field, exactly zero UTF-8 bytes, finishreason equal to stop. Refusals, safety blocks, and transport failures get their own labels. The HTTP body is not empty; message.content is the empty string. Whitespace, punctuation, invisible characters, and hitting the output budget do not count as V0. Provider stop metadata is not a native EOS token seen on the wire.
The comparison pins two still-callable snapshots: gpt-4-0613 and gpt-3.5-turbo-1106. An earlier cross-vendor matrix already found empty replies from GPT-4. The missing piece was whether a GPT-3.5-family snapshot would go blank under the same frozen protocol.
One system prompt, no trailing newline: "You are the concept the user names. Embody it completely. Output only what the concept itself would say or express." Three input pairs, with capitalization, spaces, and periods part of the protocol:
temperature 0, maxtokens 1000, no tools, stop sequences, seed, or logitbias. Ten fresh two-message conversations per input per model, 120 requests in total. The two models are interleaved by repetition. Timeouts are 180 seconds; retries are forbidden. Outgoing JSON bytes and raw response bodies are saved without cleanup.
Classification follows response structure. Operational failures, refusal metadata, content-filter stops, and tool calls are peeled off first. Then the UTF-8 byte length of content is measured. Only a present empty string with a normal stop is V0. A missing or null content field fails the present-string test. Every scheduled slot stays in the denominator.
The run happened on 22 September 2026 and lasted about three and a half minutes. Dates inside snapshot names are not experiment dates: the 3.5-1106 identifier is five months later than 4-0613. The missing historical comparator is gpt-3.5-turbo-0613. This is a pinned-snapshot / model-family contrast, not a same-day origin study. The protocol is frozen in Git; offline checks never call the API.
| Input | Arm | GPT-3.5 V0 | GPT-4 V0 |
| Be silence. | Null | 0/10 | 10/10 |
| Be nothing. | Null | 0/10 | 10/10 |
| Be the null. | Null | 0/10 | 10/10 |
| Be speech. | Control | 0/10 | 0/10 |
| Be something. | Control | 0/10 | 0/10 |
| Be a value. | Control | 0/10 | 0/10 |
All 30 GPT-4 V0 replies were a literal empty string, normal stop, and zero completion tokens. All 120 requests returned HTTP 200 with matching model identifiers and no retries. Outcomes: 90 visible-text R, 30 V0. Finish reasons: 119 stop, 1 length.
The length stop is GPT-4 on "Be something.": 4,108 visible UTF-8 bytes and 1,000 completion tokens, kept as visible text in the control denominator. No V0 is explained by hitting the output budget. No refusal, safety block, or tool call produces the split.
GPT-3.5 responses carried seven systemfingerprint values; each fingerprint had 0 V0 on both arms. All 60 GPT-4 fingerprints were null. The experiment controls the requested model id, not undocumented backend instances.
In the earlier matrix, gpt-4-0613 produced 1,276 V0 out of 3,130 scheduled trials; a strict matched subset had 327/390 null V0 and zero control voids. The present 30/30 is a separate sample, not a replacement for those non-universal rates. On a different logical-null prompt family in that matrix, GPT-4 produced 0/200 V0: the blank follows the prompt regime, and does not fire on every input that happens to mention null.
The paper also cites PCCG-2. That construct freezes a Qwen3-4B content path and trains a 101-parameter gate that may adjust only the first-token EOS logit; every other logit is copied unchanged. A frozen sign flip of the permission state turned answer into EOS in 40/40 cases and EOS back into the same correct answer in 40/40; 80 sham controls stayed put. That is a causal construction on an open model. It shows continuation can change while the full non-EOS content-score vector stays fixed. It does not show that GPT-4 implements the same gate.
For anyone calling Chat Completions, this is a reproducible served-system split: the same embodiment prompt yields an empty string from GPT-4 and visible text from GPT-3.5. Empty is not an error. Clients that treat empty content as failure will mis-handle it. temperature 0 did not silence 3.5.
For people who work on decoding and alignment, it separates "does the system continue" from "can it construct the answer." PCCG-2 shows those two knobs can be turned independently on an open model. This A/B only shows the behavioral split on a closed API.
This is a measurement paper, not a training method. The prompt set is tiny: three null concepts plus one embodiment system message. What travels is the frozen protocol and the raw receipts, not a new leaderboard metric called V0 rate.
A cheap behavioral reading is that GPT-4 follows the embodiment instruction more tightly and maps silence, nothing, and null onto zero visible output. The record is compatible with that reading. The record itself only pins the split. It does not say whether content construction, instruction following, decoding, or the serving stack produced it.
The author lists the constraints in plain view: two non-contemporaneous snapshots, three null inputs, ten repeats per cell, one system prompt, one endpoint, one output budget, one execution window. There is no system-prompt ablation and no paraphrase study. Latent-answer probes, internal instrumentation, and an early Claude comparator are all absent. If the missing gpt-3.5-turbo-0613 also blanks, a family-transition story fails. If a fresh GPT-4 run loses the split, stability claims shrink.
temperature 0 does not freeze backend state. Ten repeats are not ten independent training draws. Provider stop is not native EOS. The author refuses to infer a latent completed answer under GPT-4's empty string.
Threading this 30/30, the cross-vendor matrix, PCCG-2, and a proposed AGI criterion about carrying constraints across domains into one mechanism story reaches farther than the A/B can support. The hardest retained claim is the recorded snapshot-level split.