Fixing ill-formed UTF-16 strings with SIMD instructions
Robert Clausecker, Daniel Lemire · 2026
Read Fixing ill-formed UTF-16 strings with SIMD instructions by Robert Clausecker, Daniel Lemire (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06349.txt.
Description
Broken Unicode text can be a security risk, and Robert Clausecker and 1 co-author speed up fixing it in a 2026 paper. UTF-16 encodes characters in one or two 16-bit units. Characters outside the basic plane need a surrogate pair, a high unit followed by a low unit, and corruption can leave them mismatched.
Languages like JavaScript must repair such ill-formed strings by replacing stray surrogates with a standard replacement character. The authors use SIMD instructions, which process many data items with one instruction, to check multiple code units in parallel. Scanning in parallel makes the repair faster and more efficient. Their software ships inside Google's V8 JavaScript engine, and thus inside several major web browsers used by billions.
Catalog record
| CREATOR | Robert Clausecker, Daniel Lemire |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2601-06349