Utilizing LLMs to Investigate the Disputed Role of Evidence in Electronic Cigarette Health Policy Formation in Australia and the UK

Damian Curran, Brian Chapman, Mike Conway · 2025

Read Utilizing LLMs to Investigate the Disputed Role of Evidence in Electronic Cigarette Health Policy Formation in Australia and the UK by Damian Curran, Brian Chapman, Mike Conway (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-06782.txt.

Introduction

Damian Curran, Brian Chapman, and Mike Conway use a language model to compare how Australia and the United Kingdom argue about e-cigarettes, in a 2025 paper. The two countries read the same broad evidence yet chose opposite paths: Australia restricts the products, while the UK allows them more freely. The team gathered 109 official policy documents from both legislative processes and trained a GPT-4 classifier to tag each sentence as stressing help or harm.

The classifier scored well against human judgment. Applied to the full set, it found Australian documents heavy with harm claims and light on helpful ones. British documents leaned the other way. The authors say the split supports the idea that the same science can be framed to opposite ends. They offer the method as a tool for studying evidence in health policy.

Catalog record

CREATORDamian Curran, Brian Chapman, Mike Conway
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE20 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-06782

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Utilizing LLMs to Investigate the Disputed Role of Evidence in Electronic Cigarette Health Policy Formation in Australia and the UK Damian Curran, Brian Chapman, Mike Conway Abstract Australia and the UK have developed contrasting approaches to the regulation of electronic cigarettes, with - broadly speaking - Australia adopting a relatively restrictive approach and the UK adopting a more permissive approach. Notably, these divergent policies were developed from the same broad evidence base. In this paper, to investigate differences in how the two jurisdictions manage and present evidence, we developed and evaluated a Large Language Model-based sentence classifier to perform automated analyses of electronic cigarette-related policy documents drawn from official Australian and UK legislative processes (109 documents in total). Specifically, we utilized GPT-4 to automatically classify sentences based on whether they contained claims that e-cigarettes were broadly helpful or harmful for public health. Our LLM-based classifier achieved an F-score of 0.9. Further, when applying the classifier to our entire sentence-level corpus, we found that Australian legislative documents show a much higher proportion of harmful statements, and a lower proportion of helpful statements compared to the expected values, with the opposite holding for the UK. In conclusion, this work utilized an LLM-based approach to provide evidence to support the contention that - drawing on the same evidence base - Australian ENDS-related policy documents emphasize the harms associated with ENDS products and UK policy documents emphasize the benefits. Further, our approach provides a starting point for using LLM-based methods to investigate the complex relationship between evidence and health policy formation. 1 Introduction & Background Australia and the United Kingdom (UK) have developed contrasting approaches to the regulation of Electronic Nicotine Delivery Systems (ENDS, also known as electronic cigarettes, e-cigs, and vapes). UK policy development has been informed by a "harm reduction" approach in which existing adult smokers are encouraged to switch from combustible tobacco to (apparently, less harmful) ENDS use ( Hawkins and Ettelt 2019 ; Smith et al. 2023 ) , while Australia has developed a more restrictive ENDS regulatory environment, motivated by concerns regarding the attractiveness of these products to young people and the risk of an unchecked market in ENDS creating a new generation of nicotine-dependent adults at risk of smoking-related harms ( Freeman et al. 2023 ; Howe et al. 2024 ) . This divergence contrasts with the previously coordinated international approach to combustible tobacco regulation in the international tobacco control community. For example the implementation of standardized health packaging ( Hawkins and Ettelt 2019 ) , the introduction of legislation stipulating a minimum age for purchasing tobacco products ( Nuyts et al. 2020 ) , and the development of tax policies designed to limit tobacco consumption ( Chaloupka et al. 2012 ) . Notably, these diverse ENDS policies — more permissive in the UK, more restrictive in Australia — have developed despite public health bodies in the two jurisdictions drawing upon similar sources of evidence ( Smith et al. 2021b ; Eisenkraft Klein et al. 2022 ) . Several attempts have been made to cast light on this anomaly using a variety of approaches, including historical analyses of tobacco control approaches in the jurisdictions ( Berridge et al. 2021 ) , citation analysis of policy documents and position papers ( Smith et al. 2021b ) , and interviews with key informants in tobacco regulatory science policy ( Smith et al. 2021a ) . Common themes that have emerged are the observation that Australian policy makers focus on the potential threats of ENDS products, while UK policy makers focus on potential benefits. Several scholars have suggested that these policy differences are not necessarily entirely evidence-driven, but rather are influenced by underlying a priori difference in policy orientations ( Morphett et al. 2023 ) . In this work, we utilize Large Language Models (LLMs) to perform automated analyses of ENDS-related policy documents drawn from Australian and UK legislative processes. LLMs are complex neural network-based models that can be guided by prompt engineering to perform language processing tasks at scale. Both commercial and open-source LLMs are now commonly used in health applications. Examples include extracting adverse drug events from social media data ( Li et al. 2025 ) , developing and evaluating synthetic clinical text for training models in situations where data — or data diversity — is limited ( Lozoya et al. 2024 ) , and performing information extraction from clinical notes ( Holgate et al. 2024 ) . However, work on utilizing LLMs to better understand health policy formation is currently nascent. Our objective was to quantitatively analyze the divergent ENDS policy positions adopted in Australia and the UK. To do this, we quantified differences in the framing of ENDS-related risks in public inquiry policy documents using an LLM-based, sentence-level classifier to label sentences as either helpful , harmful , or neither with respect to the public health effects of ENDS, evaluated this classifier on a manually annotated corpus of health policy statements, and - once validated - applied the classifier to a large collection of Australian and UK public policy documents, allowing us to identify clear differences in the interpretation of evidence between the two jurisdictions. Contributions of this work include (1) a robust empirical demonstration that Australian ENDS-related policy documents emphasize the harms associated with ENDS products and UK policy documents emphasize the benefits; and (2) a demonstration that LLMs can be utilized for work in health policy analysis, without the need to generate extensive training data as required by supervised language processing algorithms. 2 Methods 2.1 Corpora In order to explore differences in evidential assertions between Australian and UK policy documents, we prepared three corpora for analysis, consisting of 109 documents (see Supplementary Materials 1 1 1 https://maconway.github.io/ENDS_regulation.html for the corpora and source information.) The first corpus - henceforth, the Erku corpus - was originally compiled by Erku et al. 2020 , and consists of policy and position statements in Australia, the UK, and New Zealand on the subject of ENDS regulation. Given that our analysis is focused on Australia and the UK, we have excluded New Zealand-related documents. The Erku corpus consisted of 36 documents after omitting the New Zealand documents and two documents that we were unable to locate. We prepared two further corpora with a view to corroborating any findings from the Erku corpus. The second corpus - the Inquiry Documents (ID) corpus - used written submissions made to public enquiries on ENDS policy in each jurisdiction, namely the 2018 House of Commons inquiry in the UK ( House of Commons 2018 ) and the 2020 Senate inquiry in Australia ( Australian Senate 2020 ) . The third corpus - the Inquiry Transcripts (IT) corpus - consists of the official transcripts of the oral hearings from these public inquiries. Five oral hearings took place in the 2018 UK inquiry and two hearings took place as part of the 2020 Australian inquiry. In line with Erku et al. 2020 ’s ( Erku et al. 2020 ) corpus creation strategy, we included reports from (and testimony on behalf of) government agencies, health charities, professional health bodies, and research groups, and excluded individuals presenting in their personal capacity, industrial lobby groups, and commercial entities. The ID

Read the complete text