Fixing ill-formed UTF-16 strings with SIMD instructions

Robert Clausecker, Daniel Lemire · 2026

Read Fixing ill-formed UTF-16 strings with SIMD instructions by Robert Clausecker, Daniel Lemire (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2601-06349.txt.

Description

Broken Unicode text can be a security risk, and Robert Clausecker and 1 co-author speed up fixing it in a 2026 paper. UTF-16 encodes characters in one or two 16-bit units. Characters outside the basic plane need a surrogate pair, a high unit followed by a low unit, and corruption can leave them mismatched.

Languages like JavaScript must repair such ill-formed strings by replacing stray surrogates with a standard replacement character. The authors use SIMD instructions, which process many data items with one instruction, to check multiple code units in parallel. Scanning in parallel makes the repair faster and more efficient. Their software ships inside Google's V8 JavaScript engine, and thus inside several major web browsers used by billions.

Catalog record

CREATORRobert Clausecker, Daniel Lemire
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2601-06349

Browse the full library · About the Tower of Babel