The Sound of Risk: A Multimodal Physics-Informed Acoustic Model for Forecasting Market Volatility and Enhancing Market Interpretability

Xiaoliang Chen*, Xin Yu, Le Chang, Teng Jing, Jiashuai He, Ze Wang, Yangjun Luo, Xingyu Chen, Jiayue Liang, Yuchen Wang, Jiaying Xie · 2025

Read The Sound of Risk: A Multimodal Physics-Informed Acoustic Model for Forecasting Market Volatility and Enhancing Market Interpretability by Xiaoliang Chen*, Xin Yu, Le Chang, Teng Jing, Jiashuai He, Ze Wang, Yangjun Luo, Xingyu Chen, Jiayue Liang, Yuchen Wang, Jiaying Xie (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-18653.txt.

Introduction

What executives say on earnings calls is scripted, but how they sound may leak the truth. Xiaoliang Chen and a large team of co-authors built a model in 2025 that reads voices on those calls with physics-based acoustics, aiming to forecast market swings.

Their Physics-Informed Acoustic Model extracts emotion signals from raw call audio, even when the sound is clipped or distorted. Voices and words are both mapped onto three plain axes: tension, stability, and arousal. Across 1,795 calls, the combined features failed to predict which way a stock would move, yet they explained up to about forty-four percent of the swing size over the next month. The shift between the scripted talk and free questions carried much of the signal.

Catalog record

CREATORXiaoliang Chen*, Xin Yu, Le Chang, Teng Jing, Jiashuai He, Ze Wang, Yangjun Luo, Xingyu Chen, Jiayue Liang, Yuchen Wang, Jiaying Xie
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-18653

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Sound of Risk: A Multimodal Physics-Informed Acoustic Model for Forecasting Market Volatility and Enhancing Market Interpretability Xiaoliang Chen*, Xin Yu, Le Chang, Teng Jing, Jiashuai He, Ze Wang, Yangjun Luo, Xingyu Chen, Jiayue Liang, Yuchen Wang, Jiaying Xie Abstract Information asymmetry in financial markets, often amplified by strategically crafted corporate narratives, undermines the effectiveness of conventional textual analysis. We propose a novel multimodal framework for financial risk assessment that integrates textual sentiment with paralinguistic cues derived from executive vocal tract dynamics in earnings calls. Central to this framework is the Physics-Informed Acoustic Model (PIAM) , which applies nonlinear acoustics to robustly extract emotional signatures from raw teleconference sound subject to distortions such as signal clipping. Both acoustic and textual emotional states are projected onto an interpretable three-dimensional Affective State Label (ASL) space— Tension , Stability , and Arousal . Using a dataset of 1,795 earnings calls ( ≈ \approx 1,800 hours), we construct features capturing dynamic shifts in executive affect between scripted presentation and spontaneous Q&A exchanges. Our key finding reveals a pronounced divergence in predictive capacity: while multimodal features do not forecast directional stock returns, they explain up to 43.8% of the out-of-sample variance in 30-day realized volatility. Importantly, volatility predictions are strongly driven by emotional dynamics during executive transitions from scripted to spontaneous speech, particularly reduced textual stability and heightened acoustic instability from CFOs, and significant arousal variability from CEOs. An ablation study confirms that our multimodal approach substantially outperforms a financials-only baseline, underscoring the complementary contributions of acoustic and textual modalities. By decoding latent markers of uncertainty from verifiable biometric signals, our methodology provides investors and regulators a powerful tool for enhancing market interpretability and identifying hidden corporate uncertainty. 1 Introduction The efficiency and fairness of financial markets are contingent upon the transparent flow of information. Yet, a persistent asymmetry of information between corporate insiders and external stakeholders can undermine investor confidence and distort market outcomes. Quarterly earnings calls, a cornerstone of corporate disclosure, are intended to bridge this information gap. However, they often feature meticulously scripted narratives designed to manage perceptions and potentially obscure underlying risks. Consequently, prevailing analytical methods that rely solely on textual content, even those powered by advanced Large Language Models (LLMs), remain vulnerable to such narrative engineering, leaving investors at a disadvantage ( Li et al. 2023 ) . This study proposes a paradigm shift in financial risk assessment by moving beyond what is said to how it is said. We scrutinize two complementary channels: the paralinguistic properties of vocal tract dynamics and the emotional sentiment of the transcribed text. Our central postulate, grounded in dynamical systems theory and computational paralinguistics, is that the cognitive load associated with managing uncertainty or concealing information correlates with involuntary, chaotic perturbations in vocal tract dynamics ( Schuller and Batliner 2013 ) . These vocal tract dynamics are physiologically rooted and thus more difficult to consciously control than word choice. To operationalize this hypothesis, we introduce the Physics-Informed Acoustic Model (PIAM), a framework engineered for robust, real-time analysis of executive sound. The model’s unique strength lies in its ability to holistically process a single sound stream to simultaneously generate a transcript, classify vocal emotion, and detect acoustic events. This integrated, multi-output nature distinguishes it from conventional pipelines that typically handle these tasks in isolation. While the broader task of forecasting market reactions from earnings calls has been explored, our holistic, physics-informed approach represents a significant departure from prior methods that often rely on separate models for speech and text or use handcrafted acoustic features. Furthermore, its foundation in nonlinear acoustics makes it exceptionally well-suited to the complex acoustic environments of corporate communications. The very nonlinear distortions introduced by teleconferencing systems—such as signal clipping and compression artifacts—are phenomena that our physics-informed approach is inherently designed to model, allowing for more robust analysis than methods that treat such distortions as generic noise. To create a unified analytical framework, we map the discrete emotion labels—from both PIAM’s acoustic analysis and a state-of-the-art LLM applied to the transcripts—onto a predefined, interpretable three-dimensional space. We term this the Affective State Label (ASL) space, characterized by Tension, Stability, and Arousal. Leveraging a large-scale corpus of earnings calls, we extract ASL-derived features that capture not only the statistical moments of these states but also their dynamic shifts between the scripted presentation and the unscripted Q&A session. Our empirical investigation yields a key insight: emotional cues, whether from vocal or textual channels, primarily signal impending uncertainty rather than directional price movements. While these features are ineffective for forecasting Cumulative Abnormal Returns (CAR), they demonstrate potent predictive capability for future realized volatility. This finding suggests that an executive’s expressed emotional state encodes latent corporate risks that may be absent from purely semantic disclosures or historical price data. By providing a more direct and less manipulable signal of risk, this methodology offers a valuable tool for augmenting market transparency and fostering more resilient financial ecosystems. 2 Related Work Our research is situated at the intersection of financial technology, affective computing, neural networks, and acoustic signal processing. 2.1 Alternative Data in Finance The quest for an informational advantage has spurred the integration of ”alternative data” into financial modeling. Textual data from news, social media, and regulatory filings have been a primary focus, with LLMs enabling sophisticated sentiment analysis and event extraction ( Li et al. 2023 ) . However, the inherent manipulability of strategically released data highlights a need for more robust, less easily controlled information channels. Our work contributes to this search by exploring the acoustic modality alongside advanced textual analysis. We propose a dual-channel approach that leverages both a physiologically grounded signal (acoustics) and a deep semantic signal (text). This multimodal philosophy posits that combining information from different sources enhances robustness and provides a more holistic understanding. It finds parallels in fields like human-robot interaction, where multimodal inputs are important for interpreting user states accurately ( Su et al. 2023 ) . 2.2 Computational Paralinguistics and Affective Computing The field of computational paralinguistics has established that paralinguistic properties are rich carriers of information about a speaker’s emotional, cognitive, and even physiological state ( Schuller and Batliner 2013 ) . Early explorations in finance attempted to leverage these cues, demonstrating that vocal parameters could hold predictive power for market outcomes ( Mayew, Parsons, and Venkatachalam 2013 ) . However, these pioneering studies often relied on large sets of hand-crafted, Low-level Acoustic Descriptors (LLDs) such as pitch contours, intensity, and Mel-Frequency Cepstral Coefficients (MFCCs). While foundational, these methods face key limitations in real-world settings. Earnings call sound is often plagued by low bitrates, background noise, and reverberation, challenges extensively documented in speech enhancement literature ( Loizou 2007 ) . These factors, along with complexities from single-channel recordings ( Benesty, Chen, and Huang 2008 ) , often limited the efficacy of traditional LLD-based models. In contrast, we advance this line of inquiry by adopting an end-to-end deep learning approach. Our PIAM model learns relevant representations directly from raw sound, bypassing manual feature engineering and enabling it to capture subtle, high-dimensional patterns that handcrafted features might miss. 2.3 Advanced Acoustic Algorithms and Physics-Informed Models Recent breakthroughs in sound processing have been driven by self-supervised learning. Models like wav2vec

Read the complete text