The Harmonic Structure of Information Contours

Eleftheria Tsipidi, Samuel Kiegeland, Franz Nowak, Tianyang Xu, Ethan Gotlieb Wilcox, Alex Warstadt, Ryan Cotterell, Mario Giulianelli · 2025

Read The Harmonic Structure of Information Contours by Eleftheria Tsipidi, Samuel Kiegeland, Franz Nowak, Tianyang Xu, Ethan Gotlieb Wilcox, Alex Warstadt, Ryan Cotterell, Mario Giulianelli (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-03902.txt.

Introduction

Eleftheria Tsipidi and seven co-authors ask whether the flow of information in a text rises and falls in regular waves. Their 2025 paper starts from a known idea: speakers try to spread information evenly so listeners are neither bored nor swamped. In practice the rate wobbles around an average, and the team test whether the wobble itself has a beat.

Using a wave-fitting method plus a new time-scaling step, they scan texts in six languages, from English and Spanish to Basque. Steady beats show up across the board. Many of the strongest beats line up with discourse structure, such as shifts between topics or sections. The authors read this as a real pressure in language, not noise. Their tools could serve studies of style, teaching, and speech planning.

Catalog record

CREATOREleftheria Tsipidi, Samuel Kiegeland, Franz Nowak, Tianyang Xu, Ethan Gotlieb Wilcox, Alex Warstadt, Ryan Cotterell, Mario Giulianelli
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE74 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-03902

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Harmonic Structure of Information Contours Eleftheria Tsipidi, Samuel Kiegeland, Franz Nowak, Tianyang Xu, Ethan Gotlieb Wilcox, Alex Warstadt, Ryan Cotterell, Mario Giulianelli Abstract The uniform information density (UID) hypothesis proposes that speakers aim to distribute information evenly throughout a text, balancing production effort and listener comprehension difficulty. However, language typically does not maintain a strictly uniform information rate; instead, it fluctuates around a global average. These fluctuations are often explained by factors such as syntactic constraints, stylistic choices, or audience design. In this work, we explore an alternative perspective: that these fluctuations may be influenced by an implicit linguistic pressure towards periodicity, where the information rate oscillates at regular intervals, potentially across multiple frequencies simultaneously. We apply harmonic regression and introduce a novel extension called time scaling to detect and test for such periodicity in information contours. Analyzing texts in English, Spanish, German, Dutch, Basque, and Brazilian Portuguese, we find consistent evidence of periodic patterns in information rate. Many dominant frequencies align with discourse structure, suggesting these oscillations reflect meaningful linguistic organization. Beyond highlighting the connection between information rate and discourse structure, our approach offers a general framework for uncovering structural pressures at various levels of linguistic granularity. https://github.com/rycolab/harmonic-surprisal 1 Introduction Studying the rate at which speakers transmit information has been a long-standing topic of interest in linguistics and cognitive science ( Shannon 1948 ; Genzel and Charniak 2002 ; Bell et al. 2003 ; Xu and Reitter 2018 ; Giulianelli and Fernández 2021 , inter alia ) . From an information-theoretic perspective, effective communication involves striking a balance between a rate sufficiently low for the receiver to successfully decode the intended message and yet sufficiently high for the sender to reduce their effort ( Zipf 1949 ; Clark and Wilkes-Gibbs 1986 ; Aylett 1999 ; Aylett and Turk 2004 ; Gibson et al. 2019 ) . In this context, information is often quantified as Shannon surprisal, i.e., the negative log probability, of the unit being communicated within its context. As evident from visualizing surprisal values across a text—see, for example, Figure 1 for token-level surprisals estimated with a Transformer language model—information rate fluctuates harmonically throughout the discourse. To predict and explain fluctuations in the surprisal of units, prior work has examined their relationship with a unit’s position within elements of the discourse structure, such as paragraphs ( Genzel and Charniak 2003 ) , topic episodes ( Qian and Jaeger 2011 ; Xu and Reitter 2016 ) , and dialogue-specific contextual units ( Giulianelli et al. 2021 ; Maës et al. 2022 ) . While these studies independently establish links between aspects of discourse context and information rate, a comprehensive framework for investigating when and how a unit’s position within its contextual structure affects its information remains an open question. More broadly, no overarching theory yet accounts for harmonic structure in global information contours. Figure 1: Illustration of Harmonic Regression on Surprisal Contours . Surprisal contours, unit boundaries, and first-order sinusoids for the first 200 tokens from a Wall Street Journal article (document wsj_1111 in the English RST Discourse Bank). Time scaling ( Section 3.2 ) is applied according to the lengths of elementary discourse units (EDUs), sentences, and paragraphs. Here, we set the coefficients of the sinusoids to 1 for illustrative purposes. See Figure 3 and Section H.1 for realistic decompositions. One influential framework for understanding the relationship between information rate and local context is the uniform information density (UID) hypothesis ( Fenk and Fenk 1980 ; Aylett and Turk 2004 ; Levy and Jaeger 2006 ; Meister et al. 2021 ) . This hypothesis posits that, within the constraints of grammar, speakers tend to distribute information as evenly as possible. UID accounts for production choices within narrow syntactic and discourse contexts; in particular, in environments that allow for multiple alternative formulations, speakers favor those that achieve greater information uniformity ( Jaeger 2010 ; Mahowald et al. 2013 ; Torabi Asr and Demberg 2015 ) . However, when UID is stretched to larger contextual units, its explanatory power weakens. At a global level, the UID hypothesis has been taken to imply that each linguistic unit contributes a constant amount of information throughout a discourse, which corresponds to a fully rational use of the communication channel ( Genzel and Charniak 2002 ) . Empirical findings, however, show that surprisal curves over discourse are rarely static ( Xu and Reitter 2016 ; Giulianelli and Fernández 2021 ; Verma et al. 2023 ) ; instead, surprisal fluctuates within a bounded range; see also Figure 1 . This suggests the pressure for uniformity is counterbalanced by competing functional constraints on communication beyond grammaticality, which become increasingly influential in longer stretches of text and dialogue. Modulating surprisal within discourse can indeed serve various functions, such as adhering to aesthetic or stylistic conventions ( Lewis 1894 ) , maintaining listener engagement and supporting comprehension ( Cervantes and Gainer 1992 ) reducing the cognitive demands of real-time production ( Bergey and DeDeo 2024 ) , and enhancing task success in cooperative interactions ( Yee et al. 2024 ) . The way in which these global pressures shape the harmonic structure of information contours is not yet well understood. To fill this gap, this paper introduces the harmonic surprisal (HS) hypothesis. We hypothesize that surprisal contours can be globally described as a mixture of periodic trends, and that the lengths of the different periods align with structural units of varying granularity. The HS hypothesis is a refinement of the recently introduced structured context (SC) hypothesis ( Tsipidi et al. 2024 ) . Both hypotheses posit that a unit’s surprisal can be predicted from the unit’s position within the contextual structure. However, the HS hypothesis establishes a specific relationship between position and surprisal rate—one that is governed by a periodic function. To operationalize the HS hypothesis, we propose a simple modification of harmonic regression. Harmonic regression is a variant of linear regression that models dependent variables as a combination of sinusoidal components; it offers a convenient way of examining whether surprisal contours exhibit periodic trends. Our modification allows us to embed hypotheses about relevant contextual structures directly into the statistical modeling procedure. Specifically, scaling the sinusoidal predictors by the length of a given structural unit, we are able to test whether the periodic trends of surprisal contours align with the boundaries of that unit. In doing so, this approach enables us to uncover interpretable harmonic structures that reflect the underlying contextual organization. Building on Tsipidi et al. 2024 ’s ( Tsipidi et al. 2024 ) findings, we focus our analysis on sentences, paragraphs, and elementary units of the rhetorical discourse structure ( Mann and Thompson 1988 ) . However, the flexibility of the proposed harmonic regression approach also allows us to observe the role of smaller units, whose influence on surprisal fluctuations is modeled by sinusoidal components with higher frequencies. Our analyses on English, Spanish, German, Dutch, Basque, and Brazilian Portuguese texts provide consistent evidence for periodicity in surprisal contours. This evidence for periodicity is particularly pronounced when we time-scale predictors to align with the boundaries of elementary discourse units (EDUs), with first-order sinusoids—those exactly corresponding to the EDU spans—having the highest amplitude. Overall, our findings indicate

Read the complete text