SEPARATING DOCUMENTS BY LEXICAL REPETITION VIA YULE'S K AND K-MEANS: VALIDATION THROUGH MONTE CARLO SIMULATION
Raquel Romes Linhares, Regis Nunes Vargas · 2026
Read SEPARATING DOCUMENTS BY LEXICAL REPETITION VIA YULE'S K AND K-MEANS: VALIDATION THROUGH MONTE CARLO SIMULATION by Raquel Romes Linhares, Regis Nunes Vargas (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/openalex-w7206213381.txt.
Description
Raquel Romes Linhares and Regis Nunes Vargas present a simple method for sorting documents by how much they repeat their vocabulary. Their tool is Yule's K, a classic measure of how concentrated a text's word use is, paired with a clustering method that splits a set of documents into two groups. To test the idea, they ran thousands of simulated document sets with known mixes of the two kinds.
Across the simulations, the method recovered the true mix with high accuracy, correctly classifying nearly ninety-nine of every hundred documents on average. Errors grew only at the extremes, where one group was very small and the two kinds of text were hard to tell apart. The result is a checked, low-cost technique for researchers who need to separate repetitive writing from varied writing at scale.
Catalog record
| CREATOR | Raquel Romes Linhares, Regis Nunes Vargas |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | OpenAlex / Revista Ibero-Americana de Humanidades, Ciências e Educação |