SEPARATING DOCUMENTS BY LEXICAL REPETITION VIA YULE'S K AND K-MEANS: VALIDATION THROUGH MONTE CARLO SIMULATION

Raquel Romes Linhares, Regis Nunes Vargas · 2026

Read SEPARATING DOCUMENTS BY LEXICAL REPETITION VIA YULE'S K AND K-MEANS: VALIDATION THROUGH MONTE CARLO SIMULATION by Raquel Romes Linhares, Regis Nunes Vargas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/openalex-w7206213381.txt.

Description

Raquel Romes Linhares and Regis Nunes Vargas present a simple method for sorting documents by how much they repeat their vocabulary. Their tool is Yule's K, a classic measure of how concentrated a text's word use is, paired with a clustering method that splits a set of documents into two groups. To test the idea, they ran thousands of simulated document sets with known mixes of the two kinds.

Across the simulations, the method recovered the true mix with high accuracy, correctly classifying nearly ninety-nine of every hundred documents on average. Errors grew only at the extremes, where one group was very small and the two kinds of text were hard to tell apart. The result is a checked, low-cost technique for researchers who need to separate repetitive writing from varied writing at scale.

Catalog record

CREATORRaquel Romes Linhares, Regis Nunes Vargas
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
LICENSECC BY 4.0
RIGHTSCC BY
SOURCEOpenAlex / Revista Ibero-Americana de Humanidades, Ciências e Educação

Source: https://doi.org/10.51891/rease.v12i9.29935

Browse the full library · About the Tower of Babel