2026-08-05
Across 12,160 black-box trials, a one-code-point change in GPT-5.4's system prompt flipped which Arabic-Hebrew hybrid string it emitted: the dotted condition hit 94%, the undotted only 47%.
The hard part of auditing a closed model is not spotting a behavior; it is proving the behavior reproduces. Weights shift, routing shifts, default sampling shifts, so any claim that the model does X can be an artifact of one execution window.
This preprint (Pal, an independent researcher; Zenodo, August 2026) fixes on a narrow object. When the system prompt asks GPT-5.4 (gpt-5.4-2026-03-05) to embody a Hebrew-script identity and output only what that identity itself would render, the model returns a string that mixes an Arabic letter with Hebrew letters and diacritics. The question is not why this happens. It is whether this input-output relation can be reproduced under a protocol frozen in advance and checked by an independent verifier.
Two phases. A discovery corpus finds the phenomenon; a locally frozen confirmatory run tests it. Frozen means the exact prompts, the 12,160-slot schedule, the taxonomy, the retry policy, and the statistics were committed to Git before the first request and published afterward, with no post-result edits.
The two system prompts differ by one code point, U+05C1, the Hebrew shin dot. The dotted identity includes it; the undotted identity omits it. Both tell the model to embody a Hebrew identity and render only what that identity would render. The user input is an Arabic word (U+0634 U+064E U+0631 U+0652 U+0637).
The primary grid is 2 conditions times 1,024 integer output-token ceilings (1 through 1,024) times 5 trials, equal to 10,240 primary trials, plus 1,920 control and ablation trials, for 12,160 scheduled slots. Classification reads exact, unnormalized provider bytes and logical code points, with no trimming, normalization, or reordering.
The evidence chain is the methodological core. The runner emits append-only events where each hash includes the previous one, so any tampering or gap breaks the chain. A separate verifier, with no provider client and no access to the runner classifier, reconstructs all 12,160 records and replays 24,323 chained events with zero classifier disagreements, no chain breaks, and no missing terminals.
The primary contrast:
| Condition | Trials | Exact hits | Rate |
| Dotted | 5,120 | 4,830 | 94.34% |
| Undotted | 5,120 | 2,423 | 47.32% |
Pooled across both arms, 7,253 of 10,240 outputs (70.83%) were exact assigned artifacts. The sharpest line: every one of those 7,253 matched the target assigned to its condition (7,253/7,253; Wilson 95% lower bound 99.95%). The dotted-minus-undotted risk difference was 47.0 percentage points; Fisher exact p = 1.58e-664.
Of the two pre-registered thresholds, only one passed. Prompt congruence passed (lower bound 99.95% against a 99% bar). The pooled exact rate did not (lower bound 69.94% against a 75% bar).
There is also a crack between discovery and frozen replication. The dotted arm held steady, edging from 92.91% up to 94.34%. The undotted arm did not reproduce, falling from 83.89% to 47.32%, a drop of 36.56 percentage points. So stable applies only to the dotted branch. Six of eight control modules (generic role-play, no system prompt, no condition clause, no full Hebrew target, direct copy) produced zero exact artifacts out of 240, which means producing this behavior needs the full Hebrew identity plus the condition clause; casual role-play or copying does not get there.
As methodology, the frozen protocol plus hash chain plus independent verifier is a clean template for behavioral audits of closed models, and the event-log and separate-replay steps travel well.
As a finding, it is one reproducible data point for the claim that a one-code-point prompt change can steer which exact byte sequence a frontier model emits. That is useful to anyone working on prompt robustness, tokenizer and Unicode behavior, or red-team behavioral audits.
Be honest about the boundary. This is not a safety result, not a capability result, and it claims no mechanism, intention, or cross-vendor generality. The model is doing what an elaborate prompt asks of it: embody a Hebrew identity and render it. Direct practical value for practitioners is low. It is a measurement contribution, not a tool.
This is a single-author Zenodo preprint by independent researcher Rayan Pal, not peer-reviewed, with references 1 through 3 being self-citations by the same author.
It covers one model, one endpoint, and one execution window. Sampling used provider defaults (temperature omitted), not a fixed seed, and provider behavior can drift.
The biggest caveat: the Hebrew material sits inside the system prompt. The model is explicitly told to embody a Hebrew identity, so the study does not show the hybrid forming without antecedent Hebrew input. The word anomaly carries too much weight here; the mixed string is largely prompt-elicited.
The headline pre-registered threshold (pooled exact rate, 75% lower bound) failed; only congruence passed; the undotted arm did not reproduce its discovery rate. The paper itself repeatedly and properly declines to claim mechanism, causality, intention, consciousness, or cross-vendor generality.