CharBench : Evaluating the Role of Tokenization in Character-Level Tasks

Omri Uzan, Yuval Pinter · 2025

Read CharBench : Evaluating the Role of Tokenization in Character-Level Tasks by Omri Uzan, Yuval Pinter (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-02591.txt.

Description

Omri Uzan and Yuval Pinter build a large test of letter level skills in language models, in a paper from 2025. Tasks such as counting letters in a word still trip top models. A folk claim blames the way models chop words into chunks, yet past checks clash on whether the chops are at fault.

The new test dwarfs past sets in size and covers a wide range of letter tasks. Mean scores sit well under half right, and some tasks fall near a third. For count tasks, word length and true count matter more than chunk traits. For place tasks, long chunks that hold the target letter do hurt scores. The authors share the test as a base for later gains.

Catalog record

CREATOROmri Uzan, Yuval Pinter
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-02591

Browse the full library · About the Tower of Babel