Measuring and Analyzing Subjective Uncertainty in Scientific Communications
Grace Shao, Jamshid Sourati * · 2025
Read Measuring and Analyzing Subjective Uncertainty in Scientific Communications by Grace Shao, Jamshid Sourati * (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-21114.txt.
Introduction
The authors measure in 2025 how strongly scientists hedge their claims in writing, and how that softness varies across fields. Findings are usually reported with hard statistics like error bars. Yet the words around the numbers carry a second, subjective layer of doubt, shaped by each author's style.
That word level doubt is known to sway public audiences. The team turned instead to scientists themselves, scoring hedging language across disciplines. Levels differed markedly from field to field. Such differences can distort how results travel between fields, since a firm tone in one field reads as doubt in another. The work offers a way to measure the tone and compare it fairly. Journals and press offices could use it to keep claims honest.
Catalog record
| CREATOR | Grace Shao, Jamshid Sourati * |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 50 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-21114
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Measuring and Analyzing Subjective Uncertainty in Scientific Communications Grace Shao, Jamshid Sourati * Abstract Uncertainty of scientific findings are typically reported through statistical metrics such as p p -values, confidence intervals, etc. The magnitude of this objective uncertainty is reflected in the language used by the authors to report their findings primarily through expressions carrying uncertainty-inducing terms or phrases. This language uncertainty is a subjective concept and is highly dependent on the writing style of the authors. There is evidence that such subjective uncertainty influences the impact of science on public audience. In this work, we turned our focus to scientists themselves, and measured/analyzed the subjective uncertainty and its impact within scientific communities across different disciplines. We showed that the level of this type of uncertainty varies significantly across different fields, years of publication and geographical locations. We also studied the correlation between subjective uncertainty and several bibliographical metrics, such as number/gender of authors, centrality of the field’s community, citation count, etc. The underlying patterns identified in this work are useful in identification and documentation of linguistic norms in scientific communication in different communities/societies. keywords subjective language uncertainty; natural language processing; scientific communications † † firstpage: 1 † † volume: 1 † † issue: 1 † † articlenumber: 0 † † year: 2024 † † copyright-year: 2024 † † hreflink: https://doi.org/ † † authornames: Grace Shao and Jamshid Sourati † † address: 1 University of Chicago, Department of Sociology (Computational Social Sciences) 2 DePaul University, School of Computing † † corresponding: Correspondence: [email protected] † † reftitle: References 1 Introduction Traditionally, scientific theories were taken as absolute claims with no subjective uncertainty around them. Today, we recognize that our scientific models are merely computational approximations of reality and, therefore, are taken within a shade of uncertainty as opposed to absolute assertions Kampourakis and McCain (2019) . Uncertainty of findings could be either obtained through the statistical analysis reported in the manuscripts through p -values, confidence intervals, etc. Böschen (2021) ; Kalmbach et al. (2023) , and/or directly based on internal structure of the findings themselves Yu et al. (2022) ; Svensson et al. (2024) 1 1 1 Such quantifiable objective uncertainty is sometimes called “risk” against the non-quantifiable lack of confidence that is simply called “uncertainty” Knight (1921) . . The formulations used to measure this type of quantifiable uncertainty can be derived independently of the scientists who conducted the work, which is why it is regarded as objective uncertainty. The degree of objective uncertainty is typically reflected in the language used by the authors to convey their findings primarily through expressions that incorporate uncertainty-inducing terms or phrases. While objective uncertainty can be reported through statistical metrics independent of the authors’ writing style, the verbal uncertainty is a subjective concept depending on how individual writers communicate their findings Dhami and Wallsten (2005) . Being able to properly interpret subjective uncertainty within the language can be helpful in the absence of reported statistical uncertainty analysis. Prior research provided evidence that the level of subjective uncertainty influences the impact of science on public audience. As an example, there have been studies showing destructive effect of high uncertainty on public trustworthiness Osman et al. (2018) , slowing down the dissemination of scientific results through social media Stavrova et al. (2024) . On the other hand, it is suggested that very high certainty can damage the authors’ reputation Dries et al. (2024) . In this paper, we switched our focus on the effect of the way scientists incorporate uncertainty in their language on their peer researchers that are potentially from other fields. Effectively communicating subjective uncertainty with respect to scientific findings is essential not only for the public audience but also for peer scientists and academicians. Understanding how to interpret findings of other researchers, especially across different disciplines, is vital for efficient knowledge exchange and interdisciplinary collaboration. This understanding will be improved by gaining insight into uncertainty-relevant language norms in different fields. It is suggested that academic institutions train researchers to effectively communicate their uncertainty to ensure adherence to the highest academic standards when imparting their findings Morgan (2024) . A question that arises is whether we have a unique set of standards for language use throughout science. Is a statement regarded as an uncertain assertion in ”Physics” also perceived as an uncertain claim in ”Sociology” (disregarding the contents), and vice versa? There is evidence from previous studies implying that different aspects of language norms differ among different scientific disciplines Kjersti et al. (2006) . There is no standard English to which the submitted scientific works are compared to in the review process, apart from a typical English grammar check. As a result, the use of English in scientific publications can significantly vary across different fields, different countries and even different times Netzel et al. (2003) . Such variations can lead to misinterpretations for both human and automated readers. One approach to mitigating this issue is to identify and document linguistic norms in scientific communication of different communities/societies. There are studies analyzing different linguistic aspects of academic papers including language informality features Hyland and Jiang (2017) , positive words Vinkers et al. (2015) ; Edlinger et al. (2023) and nominalizations Biber and Gray (2021) . Additionally, given that not all papers explicitly report objective uncertainty measurements with their findings, measuring subjective uncertainty, though noisy and subjective, can help us better calibrate and finetune our text-based objective uncertainty estimators Ningrum et al. (2025) . Here, we focused on the subjective uncertainty aspect of science communication along three dimensions: disciplines, time of publications and geographical location of institute affiliations. The topic-wise and geographical differences can complicate interdisciplinary communications posing a barrier to collaborative efforts. Whereas the temporal variations can pose difficulty when fully comprehending the studies published by previous generations of researchers. In this paper, we studied the uncertainty language norm in relationship to a range of bibliographical metrics. Our research in this work involved measuring and analyzing verbal uncertainty present in scientific articles irrespective of the underlying source of lack of confidence that may have caused the uncertainty. Technically speaking, there have been at least three types of methodologies used for quantifying subjective uncertainty: machine learning models that are fine-tuned specifically towards measuring uncertainty of any given sentence Prieto et al. (2020) ; Pei and Jurgens (2021) ; machine learning models that are trained for another purpose but whose embeddings could be used for measuring subjective confidence in a sentence; and finally approaches that have no learning parameters, e.g., algorithms based on hedging and modality detection Hyland (1996) ; Vincze (2013) ; Pyatkin et al. (2021) . In our work, we first made a comparison between the available uncertainty measurement tools and used the one with the most reliable result based on our own annotated dataset. 2 Materials and Methods 2.1 Dataset The main dataset used for this study consisted of a subset of Microsoft Academic Graph (MAG) Sinha et al. (2015) , a database of millions of scientific publications up to the end of year 2021. With each publication is saved its title, abstract, author names and more metadata (e.g., publication year, author affiliations, etc.). We chose to work with publications in 10 disciplines listed