An Unsupervised Information-Theoretic Approach to Identifying Formulaic Clusters in Textual Data
Gideon Yoffe, Yair Segev, Barak Sober · 2025
Read An Unsupervised Information-Theoretic Approach to Identifying Formulaic Clusters in Textual Data by Gideon Yoffe, Yair Segev, Barak Sober (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-07303.txt.
Introduction
Gideon Yoffe, Yair Segev, and Barak Sober hunt formulaic passages in old texts with an information-based algorithm, in a paper from two thousand twenty-five. Formulaic writing repeats set phrases under tight rules, which lowers its information content against freer prose. Finding such blocks without labels is hard in small, rich data.
Their method weighs self-information across a text and splits it into structured clusters. A continuous form of the measure keeps it valid for neural text embeddings too. Tested on proposed author layers of the Hebrew Bible, the clusters matched stylistic strata. The frame offers a yardstick for how layered texts were built.
Catalog record
| CREATOR | Gideon Yoffe, Yair Segev, Barak Sober |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 94 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-07303
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Unsupervised Information-Theoretic Approach to Identifying Formulaic Clusters in Textual Data Gideon Yoffe, Yair Segev, Barak Sober Abstract Texts, whether literary or historical, exhibit structural and stylistic patterns shaped by their purpose, authorship, and cultural context. Formulaic texts, which are characterized by repetition and constrained expression, tend to differ in their information content (as defined by Shannon) compared to more dynamic compositions. Identifying such patterns in historical documents, particularly multi-author texts like the Hebrew Bible, provides insights into their origins, purpose, and transmission. This study aims to identify formulaic clusters: sections exhibiting systematic repetition and structural constraints, by analyzing recurring phrases, syntactic structures, and stylistic markers. However, distinguishing formulaic from non-formulaic elements in an unsupervised manner poses a computational challenge, especially in high-dimensional, sample-poor data sets where patterns must be inferred without predefined labels. To address this, we develop an information-theoretic algorithm that uses weighted self-information distributions to recover structured partitions in text. The resulting clusters are interpreted from their self-information profiles and characteristic recurring features. By extending classical discrete self-information measures to a continuous formulation based on differential self-information in multivariate Gaussian distributions, our method remains applicable across various textual representations, including neural embeddings under Gaussian priors. Applied to hypothesized authorial divisions in the Hebrew Bible, our approach isolates stylistic layers and provides a quantitative framework for textual stratification. This method enhances our ability to analyze compositional patterns, offering deeper insights into the literary and cultural evolution of texts shaped by complex authorship and editorial processes. keywords Clustering, Self-information, Formulaic Texts F. Author][email protected] Plain Language Summary This work presents a method for identifying formulaic clusters—segments of text that rely heavily on repetition, structural regularity, or stylistic constraint. Such clusters can be found in a wide range of contexts, including religious texts, technical manuals, oral traditions, and institutional writing, where language is shaped by convention, genre, or the need for consistency. Detecting these patterns can offer insight into how texts were composed, transmitted, and shaped by cultural forces over time. Our approach is grounded in information theory and focuses on the concept of self-information , which measures how predictable or unpredictable an observation is under a given probability model. Recurring expressions are more probable, and therefore carry lower self-information, under the cluster-conditioned model in which they are common. The method separates text units according to differences in their cluster-conditioned feature distributions; the formulaic component is then interpreted from its self-information profile and concentration of recurring features. Benchmarking against standard clustering methods on synthetic and real data demonstrates the method’s effectiveness, particularly under the sparse, high-dimensional configurations examined here. The framework uses separate discrete and Gaussian probability models appropriate to the corresponding data representations. We apply our method to three books of the Hebrew Bible: Genesis, Exodus, and Leviticus. In Genesis, we focus on the distinction between genealogical lists and surrounding narrative material. The genealogical sections are known for their rigid, repetitive structure, and our method effectively isolates them as highly formulaic. In Exodus, we examine the division between Priestly (P) and non-Priestly texts, with the Priestly material comprising legal and cultic instructions. The method reliably separates these layers in line with traditional source-critical scholarship. In Leviticus, we test whether the method can distinguish between the Priestly source and the Holiness code (H), two stylistically distinct strata. The results show that subtle differences in recurring expressions and structural regularity allow for a clear separation between them under appropriate conditions. This work offers a flexible, information-theoretic framework for analyzing the internal structure of texts. It is well-suited to the study of layered, edited, or composite documents, where patterns of repetition and constraint signal deeper organizational principles. The method is broadly applicable to literary, historical, legal, or technical corpora, wherever formulaic language plays a role in shaping textual form. 1 Introduction An intriguing challenge in textual analysis is distinguishing formulaic layers—those characterized by recurring patterns, motifs, or stylistic conventions—from non-formulaic ones. In this context, a formulaic element is defined as a word or a sequence thereof, syntactic or grammatical structures, or thematic patterns that recur systematically within a corpus, either due to conventionalized linguistic usage, genre-specific constraints, or cultural transmission mechanisms. Formulaic expressions can function as structural scaffolds within texts, shaping their composition and reflecting underlying conventions of specific literary traditions, genres, or modes of authorship (e.g., Read and Nation, 2008 ; Paquot and Granger, 2012 ; Wood, 2019 ) . Beyond their structural significance, analyzing formulaic patterns provides insights into the dynamic interplay between creativity and constraint in language, illuminating the socio-cultural and cognitive forces that have influenced textual transmission and stylistic evolution (e.g., Magoun Jr, 1953 ; Jensen, 1980 ) . The importance of formulaicity is particularly pronounced in historical texts, where layers of composition often reflect complex authorship and redaction processes. Many historical documents, created and transmitted over centuries, are products of collective memory and institutional influence. As such, these texts often contain formulaic structures that encode cultural norms, narrative conventions, and theological or political ideologies (e.g., Knohl, 2007 ; Polak, 2017 ) . Understanding these layers can reveal how texts were constructed, transmitted, and adapted over time (e.g., Gitay, 1980 ; Polak, 2006 ; Coleman, 2019 ) . Among historical texts, the Hebrew Bible provides a compelling case study. As a composite text shaped by centuries of authorship, redaction, and transmission, the Hebrew Bible embodies a rich interplay of formulaic and non-formulaic elements. Its stylistic diversity reflects contributions from different schools of authorship, genres, and literary motifs, ranging from tightly structured legal codes to more fluid poetic and narrative traditions ( Wärters, 1976 ; Shectman and Baden, 2009 ; Stipp, 2017 ) . Detection and analysis of formulaic patterns within the Hebrew Bible offer a pathway to better understanding its compositional history, shedding light on how diverse voices and traditions coalesced into the canonical form. Understanding the distinction between formulaic and non-formulaic layers requires rigorous methods for quantifying stylistic and structural patterns across literary feature sets. One promising metric for addressing these challenges is entropy, a concept from information theory that measures the unpredictability or randomness of a dataset. Initially introduced by Rudolf Clausius in 1865 as a measure of energy dispersal in a system, and subsequently formalized statistically by Shannon (1948) , entropy has found applications in diverse fields, including thermodynamics, cryptography, and data compression. In the context of textual analysis, entropy provides a framework for quantifying the variability and regularity inherent in linguistic structures. Formally, given a discrete probability distribution P=\{p_{1},p_{2},\dots,p_{n}\} over n possible elements, Shannon entropy is defined as: H(P)=-\sum_{i=1}^{n}p_{i}\log p_{i}, (1) where p_{i} represents the probability of the i th element. Higher entropy values indicate greater variability, while lower entropy values correspond to constrained, repetitive structures, making entropy a powerful tool for distinguishing formulaic clusters from more unpredictable distributions. The relevance of entropy to textual analysis lies in its ability to capture the degree of predictability of literary or linguistic features within a sequence, such as a sentence, paragraph, or chapter. Texts with high formulaicity, such as legal codes or liturgical compositions, are likely to exhibit low entropy, such that text units within them resemble one another, for example, in vocabulary or grammatical structure. Conversely, more creative or extemporaneous texts often exhibit higher entropy, reflecting