Business Entity Entropy

Adam McCabe, Matthew H. Chequers · 2025

Read Business Entity Entropy by Adam McCabe, Matthew H. Chequers (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-07106.txt.

Introduction

Adam McCabe and Matthew Chequers coin in a 2025 paper entity entropy, a measure of how scattered a topic's knowledge sits across company documents. Firms generate webs of linked content that language models struggle to search well. Some entities are simply harder to retrieve than others, and the gap was unmeasured.

Their score counts how an entity's facts spread across the document pile. On a large enterprise corpus, entropy runs heavy tailed across entities. Bigger entities tend to carry higher entropy, and content classes show set patterns. A generative model in the diffusion style is offered to explain those patterns. The upshot is entity aware retrieval for the hard cases, not a blanket fix.

Catalog record

CREATORAdam McCabe, Matthew H. Chequers
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE56 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-07106

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Business Entity Entropy Adam McCabe, Matthew H. Chequers Abstract Organizations generate vast amounts of interconnected content across various platforms. While language models enable sophisticated reasoning for use in business applications, retrieving and contextualizing information from organizational memory remains challenging. We explore this challenge through the lens of entropy, proposing a measure of entity entropy to quantify the distribution of an entity’s knowledge across documents as well as a novel generative model inspired by diffusion models in order to provide an explanation for observed behaviours. Empirical analysis on a large-scale enterprise corpus reveals heavy-tailed entropy distributions, a correlation between entity size and entropy, and category-specific entropy patterns. These findings suggest that not all entities are equally retrievable, motivating the need for entity-centric retrieval or pre-processing strategies for a subset of, but not all, entities. We discuss practical implications and theoretical models to guide the design of more efficient knowledge retrieval systems. 1 Motivation and Intuition Organizations today generate a continuous stream of content – documents, meeting transcripts, emails, code repositories, database entries, etc – accumulating into vast, interconnected knowledge bases. Within this evolving digital environment, organizations face the fundamental challenge of knowledge creation and management [ 10 ] . Leaders and decision-makers increasingly rely on natural language processing and large language models (LLMs) to surface relevant insights and inform strategic choices. While powerful language models now enable sophisticated reasoning and summarization through retrieval-augmented generation (RAG) techniques [ 8 ] , a crucial challenge remains: effectively retrieving and contextualizing the right information from the sprawling organizational memory. Standard RAG pipelines assume that relevant knowledge can be found by pulling a handful of pertinent documents and feeding them to a language model. This approach is effective when the necessary facts cluster together in a small number of documents. However, not all business entities or topics are neatly confined. Some are indeed “low-entropy” entities – key concepts, individuals, products, or initiatives whose essential information resides in just one or two documents. Retrieving their full context is straightforward. Yet, other entities display “high-entropy” characteristics – they are diffusely mentioned across dozens or even hundreds of documents, with no single source containing a majority of their facts. For such entities, simple document retrieval hits a scalability wall: either the system omits critical information or must include a large swath of potentially irrelevant material, leading to inefficiency and incomplete answers. This pattern hints at an underlying structure in how organizational knowledge distributes itself. Some entities remain tightly bound to a few canonical sources; others fragment unpredictably across the knowledge base, defying simple retrieval strategies. Drawing on information theory, we conceptualize this dispersion as an “entropic” property of the entity’s knowledge footprint. Building on foundational work in information theory [ 13 , 4 ] , we propose that the varying degrees of entity “spread” can be rigorously measured and analyzed. Why entropy? In statistical physics, entropy measures the number of possible ‘micro-state’ arrangements that give rise to the same ‘macro-state’ observation. We can apply this concept to our context in that many different arrangements of facts across documents can lead to the same overall knowledge about an entity. When facts about an entity are concentrated in a single source, there are few possible arrangements, leading to low entropy. However, when facts are evenly spread across hundreds of documents, there are many possible arrangements, resulting in high entropy. This physical interpretation maps elegantly to information theory, where entropy measures uncertainty about which message – or in our case, which document – will appear next. Both interpretations are equivalent [ 13 ] , and capture the same fundamental idea: when information is highly dispersed, the system faces greater uncertainty in knowing which documents to retrieve or how to combine them efficiently. For the bulk of this paper, we will use the information theory conceptualization, but at times may draw on intuition from the physical interpretation. This entropic model of context distribution opens several avenues for both practical system design and theoretical inquiry. Theoretically we find that entropy can be used to understand a generative model describing a feedback mechanism helping to grow entropy early on based on initial importance, while tapering off as entropy saturates to the theoretical maximum (Section 4 ). From a practical standpoint, quantifying entropy allows architects of AI decision-support platforms to distinguish between entities that can be handled by simple retrieval methods and those that require more sophisticated, entity-centric pre-processing or more sophisticated retrieval strategies (Section 5 ). From a theoretical perspective, understanding why some entities become “entropic” and others remain “low-entropy” could shed light on the underlying generative processes in organizational knowledge formation, providing practical identification algorithms for organizations. Moreover, introducing an entropy framework sets the stage for deeper questions about how knowledge evolves over time, how it compares across different organizations or domains, and whether certain types of entities inherently resist stable concentration (Section 3 ). By quantifying these distributions, we not only provide a new lens to analyze organizational memory but also lay the groundwork for a richer theoretical understanding of how human-created documents shape – and are shaped by – contextual complexity. The application of entropy to named entities is not novel. Early work by [ 3 ] demonstrated the effectiveness of maximum entropy frameworks for named entity recognition (NER), using entropy to model the uncertainty in entity classification across different contexts. However, our work departs from this traditional use of entropy in NER tasks. Rather than applying entropy to improve entity detection and classification, we propose a novel entropy measure that quantifies how entity knowledge distributes itself across an organization’s document corpus. This work aims to bridge the gap between the engineering pragmatics of RAG and a richer, more mathematically grounded understanding of enterprise knowledge distribution. In the following sections, we will formalize intuitions related to entropy in Section 2 , present empirical findings in Section 3 , propose theoretical models that explain the observed patterns in Section 4 , and discuss practical applications in Section 5 . Finally, in Section 6 we highlight our conclusions. Throughout this paper we will refer to the Appendix in Section 7 for supporting material. 2 Defining the Entropic Measures To systematically capture the intuitive notion of “contextual sprawl”, we introduce a formal measure of entropy inspired by information theory. Just as Shannon entropy quantifies the uncertainty in a probability distribution over possible messages [ 13 ] , we apply a similar framework to measure how evenly facts about a given entity are distributed across a set of documents. 2.1 Setting and Notation Consider an organization’s corpus of documents, \mathcal{D}=\{d_{1},d_{2},\ldots,d_{N}\} , where each d_{i} is a text artifact (e.g., a meeting transcript, a report, a GitHub issue, etc, or chunks of such) produced or archived by the business. Within this corpus, we focus on a particular physical or conceptual entity [ 12 ] E , which could be a product, a key individual, an initiative, or a concept relevant to the business. Our goal is to understand how the knowledge associated with E is distributed across \mathcal{D} . For each document d\in\mathcal{D} , let f_{E}(d) represent the number of distinct facts about the entity E that appear in d . Facts represent single pieces of

Read the complete text