2026-08-26
A SIMD kernel that replaces unpaired UTF-16 surrogates with U+FFFD reaches 18.9 GB/s on Apple M4 (about 9× V8 scalar) and 7.5 GB/s on Ice Lake; it ships in Chromium and Node.js 25.
JavaScript, Java, and Windows keep strings as UTF-16. Characters in the Basic Multilingual Plane (BMP, U+0000–U+FFFF) fit in one 16-bit code unit. Everything else, emoji and rarer CJK included, is a surrogate pair: a high surrogate in U+D800–U+DBFF must be followed by a low surrogate in U+DC00–U+DFFF. A lone surrogate is ill-formed UTF-16.
Those strings show up from truncation, bad transcoding, and hostile input. CPython 3.1 through 3.3 failed to update a decoder bound after its error handler ran on an unpaired surrogate, read past the buffer, and became CVE-2012-2135. The usual fix is to replace unpaired surrogates with U+FFFD. ECMAScript added String.prototype.toWellFormed for this. V8 implemented it as a scalar loop, one code unit at a time. On long strings that loop is a bandwidth tax.
Clausecker (Zuse Institute Berlin) and Lemire (TELUQ) rewrite the loop with SIMD, at least eight 16-bit code units per step. The hard part is a surrogate pair that sits across a vector boundary.
Each iteration loads two vectors offset by one code unit. The earlier vector (lookback) is tested for high surrogates; the later one (block) for low surrogates. XOR the two boolean masks. All zeros means the block is well sequenced, so copy it, or skip the store entirely when running in place. If the XOR is nonzero, mark "high not followed by low" and "low not preceded by high", then blend U+FFFD over those lanes.
Three choices keep the fast path cheap. Overlapping vectors instead of shifting inside one register kill loop-carried dependence, so the CPU can overlap iterations. Two loads per iteration beat slicing lookback out of the previous block, because modern cores retire multiple SIMD loads per cycle and the arithmetic still dominates (high operational intensity: ALU SIMD ops relative to loads and stores). The code is branchy: almost all UTF-16 is valid, so a cheap "any error?" check runs first.
Inputs shorter than one vector plus one element fall back to scalar. A leading lone low surrogate and a trailing lone high surrogate are patched by hand. The algorithm is idempotent, so a short tail is handled by one last iteration aligned to the end, overlapping the previous block.
SSE2 and AVX2 follow the generic recipe. AVX-512 switches to mask registers, 32 code units (64 bytes) at a time, and writes illegal lanes with masked stores. SVE is skipped for lack of hardware. On NEON there is no pmovmskb, and testing "is this vector all zeros?" goes through vmaxvq into a general-purpose register, which is high latency. The workaround processes four 16-unit chunks, 64 code units, then checks once. LD2 deinterleaves UTF-16 into high and low bytes; only the high byte is needed to classify a surrogate, so the element width drops from 16 bits to 8.
The C++ kernels live in simdutf. The baseline is V8's scalar implementation of the day. Inputs go up to 1,000,000 code units, with 0.1% valid surrogate pairs and either 0% or 0.1% unpaired surrogates. 100 runs, best time, about 1% error margin.
| Platform | Kernel | 0% ill-formed (GB/s) | 0.1% ill-formed (GB/s) | ins/byte |
| Apple M4 | V8 scalar | 2.2 | 2.2 | 12.0 |
| Apple M4 | NEON | 18.9 | 16.3 | 0.9 |
| Xeon Gold 6338 | V8 scalar | 1.2 | 1.2 | 13.0 |
| same box, Ice Lake AVX-512 | simdutf | 7.5 | 7.4 | 0.4 |
| same box, Haswell AVX2 | simdutf | 7.8 | 7.6 | 0.8 |
| same box, Westmere SSE | simdutf | 5.8 | 5.6 | 2.0 |
On the M4 the NEON kernel is about 9× the V8 scalar path, and instruction density falls from 12.0 per byte to 0.9. Ice Lake's AVX-512 kernel is cheaper still at 0.4 instructions per byte and 7.5 GB/s, a hair below the same box's Haswell AVX2 kernel at 7.8 GB/s. The prose calls Ice Lake the throughput winner; the table does not.
The same functions landed in Node.js 25. toWellFormed on a 1024-character random string (5% low surrogates, 5% high, rest ASCII):
| Runtime | Apple M4 | Xeon Gold 6338 |
| Node.js 24.13.0 | 2.9 GiB/s | 2.0 GiB/s |
| Node.js 25.5.0 | 16 GiB/s | 11 GiB/s |
About 5×. JavaScript overhead eats part of the 9× seen in C++. On the same string and an Apple M4, Chrome 144 reaches 16 GiB/s, Firefox 147 2.7, Safari 18.6 1.0. Chromium already ships this path.
Nothing new in Unicode semantics. The work is the engineering: toWellFormed, which browsers and Node run constantly, moves from a scalar loop to a near-bandwidth vector kernel, in place, no allocation, copy or not.
If the job is repairing ill-formed UTF-16, call simdutf. V8, Chromium, and Node 25 already do. Firefox and Safari are still in scalar territory: on that 1024-character string, Chrome 144 on an M4 is about 6× Firefox 147 and 16× Safari 18.6.
For people writing tokenizers, JSON parsers, or log pipelines, the useful artifact is the design: overlapping lookback, a validity probe before the slow path, LD2 to drop the low byte on NEON. Narrow problem. Already in production. The same group previously published SIMD UTF-8 validation and UTF transcoding; this paper is incremental on the algorithm side, not on the deployment side.
There is no limitations section. The method text states the edges: short strings go scalar; the first and last code units need scalar patches; the fast path assumes valid input. On the M4, 0.1% unpaired surrogates drop throughput from 18.9 GB/s to 16.3. Higher error rates are unmeasured.
Inputs are synthetic random strings, not the JS heap. No comparison with ICU or other SIMD libraries. SVE is omitted. simdutf also has RISC-V and LoongArch kernels; the paper only reports an M4 and a 2019 Ice Lake Xeon.
The conclusion says "up to eightfold"; the experiments section says "nearly 9 times". 18.9 / 2.2 is about 8.6. Treat the table as source of truth for Ice Lake versus Haswell.