On Large Language Models as Data Sources for Policy Deliberation on Climate Change and Sustainability

Rachel Bina, Kha Luong, Shrey Mehta, Daphne Pang, Mingjun (Kristen) Xie, Christine Chou, Steven O. Kimbrough, (and the Penn Climate Decisions Lab) · 2025

Read On Large Language Models as Data Sources for Policy Deliberation on Climate Change and Sustainability by Rachel Bina, Kha Luong, Shrey Mehta, Daphne Pang, Mingjun (Kristen) Xie, Christine Chou, Steven O. Kimbrough, (and the Penn Climate Decisions Lab) (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-05708.txt.

Introduction

City councils weighing climate policies face long lists of options and criteria, and a language model might supply first-pass scores. Rachel Bina and co-authors test that idea in this 2025 study. Their question is whether a chatbot's scores are credible enough to start a structured debate, not to settle one. The method they use ranks options against many quality-of-life criteria at once.

The team listed climate and sustainability policies that local governments in the United States actively weigh. GPT-4 scored each policy on each criterion. A standard ranking method then ordered the policies from those scores. The ranking was checked against a ranking built by informed human judgment. The chatbot's order roughly matched the human one. The authors frame the tool as a starter for deliberation, not a judge.

Catalog record

CREATORRachel Bina, Kha Luong, Shrey Mehta, Daphne Pang, Mingjun (Kristen) Xie, Christine Chou, Steven O. Kimbrough, (and the Penn Climate Decisions Lab)
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE67 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-05708

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

On Large Language Models as Data Sources for Policy Deliberation on Climate Change and Sustainability Rachel Bina, Kha Luong, Shrey Mehta, Daphne Pang, Mingjun (Kristen) Xie, Christine Chou, Steven O. Kimbrough, (and the Penn Climate Decisions Lab) Abstract We pose the research question, Can LLMs provide credible evaluation scores, suitable for constructing starter MCDM models that support commencing deliberation regarding climate and sustainability policies? In this exploratory study we i. Identify a number of interesting policy alternatives that are actively considered by local governments in the United States (and indeed around the world). ii. Identify a number of quality-of-life indicators as apt evaluation criteria for these policies. iii. Use GPT-4 to obtain evaluation scores for the policies on multiple criteria. iv. Use the TOPSIS MCDM method to rank the policies based on the obtained evaluation scores. v. Evaluate the quality and validity of the resulting ensemble of scores by comparing the TOPSIS-based policy rankings with those obtained by an informed assessment exercise. We find that GPT-4 is in rough agreement with the policy rankings of our informed assessment exercise. Hence, we conclude (always provisionally and assuming a modest level of vetting) that GPT-4 can be used as a credible input, even starting point, for subsequent deliberation processes on climate and sustainability policies. Democracy begins with conversation. —John Dewey at his 90 th birthday party 1 1 1 Lamont, C. (Ed.). (1959). Dialog on John Dewey. Horizon Press. Page 88. Only that which has no history can be defined. —Frederick Nietzsche On the Genealogy of Morals The definition of the problem, rather than its solution, will be the scarce resource in the future. —Esther Dyson (1990s) Keywords: large language models, LLM, MCDM, climate change, sustainability, biodiversity 1 Context This study occurs in the context of a larger project that is directed at, among other things, two design goals. The first of these goals is about providing deliberation tools for supporting decision making on climate and sustainability policies. Design Goal 1 (tools) . Construct tools to support deliberation on climate and sustainability policies. These tools may come in any of several forms, including knowledge of methods and practices, as well as databases, document corpora, and software. The list of potential forms is open-ended. The second motivating design goal is more specific. It constitutes a way of achieving instances that meet the first goal. The second goal is about MCDM (multi-criteria decision-making) models. These constitute a broad class of modeling methods that are designed and intended to facilitate decision-making in the presence of two or more conflicting goals and objectives. This kind of situation is rife in everyday life, for example, when we make purchasing decisions that have to trade off cost and quality. Policy decisions in climate change and sustainability are more complex, typically having more than two goals or evaluation criteria that have to be considered. The goals are typically in some degree of conflict and themselves complex. MCDM models, while long established and used, are under-employed. This is in significant part due to the fact that developing a model requires specialized expertise, which may not be available or affordable in particular cases, as well as rather intense and laborious feedback from decision makers, who are busy and often skeptical of yet-to-be-seen tools ( Keeney, 1992 ; Keeney and Raiffa, 1993 ) . This occasions our second design goal. Design Goal 2 (starter MCDM models) . Deliver to climate and sustainability policy stakeholders MCDM models (“starter MCDM models”) that are prima facie acceptable, comprehensible, and suitable for follow on deliberation, revision, and evolution. These initial models should require minimal input from stakeholders, while affording revision in light of new information and stakeholder views when available. We foresee being able to achieve Design Goal 2 and thereby being able to afford processes in which technical analysts develop prima facie valid (alias “not stupid”) starter MCDM models that may be used in public deliberations. Under extreme time constraints, such models would be usable without substantial revision for decision making. (This is not to say that the model’s assessments are simply accepted; rather, the assessments are subject to deliberation with the information available.) In the more usual case, the model is revised and evolved in light of new information and public comments. In this way, an ever-improving, presumably valid knowledge base contributes to ongoing discussions and is available whenever decisions have to be made. With these goals before us, we hypothesize that large language models (LLMs), such as GPT-4, can be instrumental in creating starter MCDM models that meet Design Goal 2 . Research Question . Can LLMs provide credible evaluation scores, suitable for constructing starter MCDM models that support commencing deliberation regarding climate and sustainability policies? This is all quite aspirational. Certainly, no single study can meet the design goals and dispositively answer the research question. This paper reports on an exploratory study that bears on the research question in the context of the two design goals. The upshot of our study is to give a positive answer to the research question. We turn now to essential framing and setup for what is to follow. * * * Table 1 is a schema or template for what we shall alternatively call an ACS ( a lternatives, evaluation c riteria, scores ) or P ( p erformance) table. Such tables are essential data objects when comparing multiple alternatives (the a_{i} policy alternatives) on multiple dimensions of value (the c_{j} evaluation criteria). The entries s_{i,j} in the table represent the evaluation scores for their associated a_{i} and c_{j} alternatives and evaluation criteria: s_{i,j} is the evaluation score for alternative a_{i} on criterion c_{j} . Policy Evaluation Criteria Alternatives c_{1} c_{2} … c_{j} … c_{n} a_{1} s_{1,1} s_{1,2} … s_{1,j} … s_{1,n} a_{2} s_{2,1} s_{2,2} … s_{2,j} … s_{2,n} ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ a_{i} s_{i,1} s_{i,2} … s_{i,j} … s_{i,n} ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ ⋮ a_{m} s_{m,1} s_{m,2} … s_{m,j} … s_{m,n} Table 1: General form—schema or template—for an ACS (alias P ) table. Given a complete performance table, P , decision makers have something definite and useful to work with in deliberating which alternatives to select for implementation. Of course, additional information may be useful and desired. Even so, an ACS ( P ) table is a fundamental requirement and starting point for serious deliberation. We emphasize starting point because modeling and the deliberation it supports should above all be seen as dynamic processes, halted only by practical considerations. Our concern here is with methods for arriving at data and models that are useful because they can contribute to supervening processes. Not final results, but results positioned for getting better results. 2 2 2 And that can do well enough if necessary. This raises the question of how ACS tables of usefully high quality can and should be constructed to support decision making in specific contexts. We are especially concerned in this study with climate and sustainability policies and with prospects for reducing the workload of scoring P tables. Specifically, this study addresses the paper’s Research Question, page Research Question , § 1 . In what follows, we a. Identify a number of interesting policy alternatives that are actively considered

Read the complete text