From 124 Million Tokens to 1,021 Neologisms: A Large-Scale Pipeline for Automatic Neologism Detection

2026

Read From 124 Million Tokens to 1,021 Neologisms: A Large-Scale Pipeline for Automatic Neologism Detection (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-06426.txt.

Description

Diego Rossini and Lonneke van der Plas build a large pipeline for finding new words automatically. A neologism is a genuine lexical innovation, but most candidate tokens are names, foreign words, or noise. Their pipeline combines rule-based filtering with language model classification, guided by grammatical and extra-grammatical word formation.

Applied to 527 million Reddit posts from 2005 to 2024, it narrows 124.6 million unique tokens down to 1,021 candidates for expert checking. Several models vote on each candidate, and manual annotation confirms 599 as genuine innovations. The code and annotated list are public, and model disagreement itself shows how hard the task is.

Catalog record

YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.06426

Browse the full library · About the Tower of Babel