Modeling Statistical Properties of Written Text

M. Ángeles Serrano, Alessandro Flammini, Filippo Menczer · 2009

Read Modeling Statistical Properties of Written Text by M. Ángeles Serrano, Alessandro Flammini, Filippo Menczer (2009) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/plos-pone-0005372.txt.

Description

Written language follows statistical laws, of which Zipf's law about word frequency is the most famous. Serrano and colleagues modeled the broader statistical properties of written text in PLOS ONE in 2009, beyond that single law.

Universal regularities in text can reveal how brains process information and how societies organize shared knowledge. Their models aim to capture several such patterns at once. The work treats language as a complex system whose numbers can be explained, not just described. If text has deep laws, then search tools, code that sums up long works. And tests that spot fake text can all be built on those laws and work better than rules of thumb.

Catalog record

CREATORM. Ángeles Serrano, Alessandro Flammini, Filippo Menczer
YEAR2009
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
LICENSECreative Commons Attribution License
RIGHTSCC BY
SOURCEPLOS ONE

Source: https://journals.plos.org/plosone/article?id=10.1371/journal.pone.0005372

Browse the full library · About the Tower of Babel