Statistical Mechanics of Semantic Compression
Tankut Can · 2025
Read Statistical Mechanics of Semantic Compression by Tankut Can (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-00612.txt.
Introduction
Tankut Can applies statistical mechanics to the problem of semantic compression, in work from 2025. Semantic compression means shortening a message while keeping its meaning, not just its bits. The right measure of damage is distance in meaning space, not a count of changed bits.
Can models meaning space as a high-dimensional vector space, where texts and images sit as points and distance between points gives difference in meaning. The search for the shortest meaning-preserving message becomes a spin glass problem, a classic system in physics. Solving it with replica theory maps out phases of compression. One sharp transition marks the emergence of paraphrases, while a smooth crossover separates quoting from summarising. Simulations with annealing and greedy methods support the picture.
Catalog record
| CREATOR | Tankut Can |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 78 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-00612
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Statistical Mechanics of Semantic Compression Tankut Can Abstract The basic problem of semantic compression is to minimize the length of a message while preserving its meaning. This differs from classical notions of compression in that the distortion is not measured directly at the level of bits, but rather in an abstract semantic space. In order to make this precise, we take inspiration from cognitive neuroscience and machine learning and model semantic space as a continuous Euclidean vector space. In such a space, stimuli like speech, images, or even ideas, are mapped to high-dimensional real vectors, and the location of these embeddings determines their meaning relative to other embeddings. This suggests that a natural metric for semantic similarity is just the Euclidean distance, which is what we use in this work. We map the optimization problem of determining the minimal-length, meaning-preserving message to a spin glass Hamiltonian and solve the resulting statistical mechanics problem using replica theory. We map out the replica symmetric phase diagram, identifying distinct phases of semantic compression: a first-order transition occurs between phases marked by the emergence of paraphrases, whereas a continuous crossover is seen from extractive to abstractive compression. We speculate on which features of the phase diagram are captured by replica symmetry, and which features change under replica symmetry breaking. We conclude by showing numerical simulations of compressions obtained by simulated annealing and greedy algorithms, and argue that while the problem of finding a meaning-preserving compression is computationally hard in the worst case, there exist efficient algorithms which achieve near optimal performance in the typical case. Preprint: APS/123-QED I Introduction Human working memory has a limited capacity, as revealed from numerous experiments using unstructured stimuli [ 1 ] . Nevertheless, we have the ability to process information on extremely long timescales, in apparent contradiction to this finite capacity. The crucial ingredient for accomplishing this is compression, often called “chunking” in cognitive science, whereby stimuli are recoded into more compact representations [ 2 ] . Compression is routinely observed in social communication. Bartlett [ 3 ] showed that when stories are transmitted between humans, they tend to become shorter and more stereotyped. Furthermore, in experiments on human memory for narratives, in which subjects read a story and are subsequently asked to retell it, there is a strong tendency to produce a compressed version of the story in the retelling, using efficient paraphrases and summaries [ 4 , 5 , 6 ] . Importantly, human communication involves lossy compression. For instance, in the narrative memory experiments described above, the verbatim text of the original story, also known as the surface structure, cannot be reconstructed from the recall of participants. If the surface structure does not seem to matter, then what is being transmitted during communication? A natural hypothesis is that meaning is the important thing, and surface structure can be sacrificed as long as meaning is kept invariant. But what is meaning? This notoriously elusive concept finds its most concrete formulation in the study of semantics, which seeks to understand how meaning arises in language [ 7 ] . Far from being a mere formal exercise, the study of semantics is central to the psychology of human memory. Short-term sentence recognition experiments show that details of the wording of a sentence are easily forgotten, while the meaning or “gist” of a sentence is kept much longer and more stably in memory [ 8 , 9 , 10 ] . This can be seen by testing paraphrases which preserve meaning, against sentence variants which change the original meaning. Measuring forgetting over longer timescales (days to months) confirms this observation, showing that memory for surface structure decays much faster than memory for semantic structure, which includes higher-level abstractions of a text that together give it meaning to an individual [ 11 , 12 ] . In short, our memory for discourse appears to be primarily semantic in nature. Therefore, while the compression involved in human communication may lose surface structure, it tends to preserve semantic structure; for this reason, we refer to this process as semantic compression . Semantic compression therefore plays a central role in human communication. Traditionally, lossy compression is the purview of rate-distortion theory, and semantic compression has been studied precisely in this context [ 13 , 14 ] . Furthermore, there have been more general theories which frame pragmatic communication between agents in terms of optimal transport [ 15 ] . While necessary, these general theoretical frameworks leave open some questions about the mechanism of compression in particular settings. For instance, what is the interplay between the structure of representations in semantic space, and the capacity for compression? To address such a question, we must try to make explicit contact with the representations that humans (and machines) make use of. In other words, we must specify the semantic distortion function. To define the distortion function, we take inspiration from both machine learning and cognitive neuroscience. Lexical items, such as words and phrases, are clearly stored in long-term memory [ 7 ] . Studying brain responses during a story listening task, researchers were able to map out what they called a semantic network in the brain [ 16 , 17 ] . This was then used to argue that semantic representations are continuously represented in brain activity in the whole cortex [ 18 ] . The idea of semantic spaces has a long history in psychology, dating back to the spreading activation theory in the late 60’s [ 19 ] , and later [ 20 ] . Similarly, Gärdenfors [ 21 ] makes the argument for continuous conceptual spaces in neural population representations. Semantic spaces were also studied in [ 22 ] , and argued to be closely related to the brain’s native instruments for representation of spatial geometry. This mirrors the ability of hippocampus to generate cognitive maps which represent abstract categories instead of spatial location [ 23 ] . The question remains: how does the brain represent the presumably high-dimensional spaces involved in our semantic knowledge base? Indeed, it was been argued that low-dimensional spaces are insufficient to describe the geometry of concepts [ 24 ] , and that complex networks [ 25 ] or high-dimensional distributed representations [ 26 ] are needed to account for semantic similarity judgements. A recent review has even argued in favor of explicit vector space representations of concepts [ 27 ] . In parallel research, language modeling in machine learning has naturally come upon the idea of using continuous vector spaces to represent word meanings. Some particularly vivid examples of this come from algorithms such as word2vec [ 28 ] or GloVe [ 29 ] , which map each word or token in a lexicon to an embedding vector which lives in a high-dimensional Euclidean space. Remarkably, meaningful relations between words, such as analogies, are found to be encoded in geometric relations between their vector embeddings. For instance, the vector sum of ’royal’ and ’man’ is close to ’king’ and ’prince’ (albeit also close to other seeming non-sequiturs). Similarly, analogies can be represented by vector addition, as with, {\bf v}({\rm duke})-{\bf v}({\rm male})\approx{\bf v}({\rm duchess})-{\bf v}({\rm female}) . More generally, contrastive representation learning is an approach in deep