Towards Automated Scoping of AI for Social Good Projects

Jacob Emmerson, Rayid Ghani, Zheyuan Ryan Shi · 2025

Read Towards Automated Scoping of AI for Social Good Projects by Jacob Emmerson, Rayid Ghani, Zheyuan Ryan Shi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-20010.txt.

Introduction

Jacob Emmerson, Rayid Ghani, and Zheyuan Ryan Shi test whether a language model can do the hardest first step of social-good AI projects: scoping the problem, in a 2025 paper. Projects that aim AI at transit, health, or wildlife problems often stall at the proposal stage, because few people combine technical and domain expertise. The authors build a Problem Scoping Agent that drafts full project proposals grounded in scientific literature and real-world knowledge.

In blind review, the agent's proposals were judged comparable to those written by human experts, and AI-based evaluations agreed. The result suggests much of the scoping labor, surveying what is known and framing a feasible project, can be automated well enough to unblock scarce expert time. The authors also document where real-world scoping still resists automation. They close with open challenges and directions for making such agents reliable partners in social-good work.

Catalog record

CREATORJacob Emmerson, Rayid Ghani, Zheyuan Ryan Shi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE40 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-20010

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Automated Scoping of AI for Social Good Projects Jacob Emmerson, Rayid Ghani, Zheyuan Ryan Shi Abstract Artificial Intelligence for Social Good (AI4SG) is an emerging effort that aims to address complex societal challenges with the powerful capabilities of AI systems. These challenges range from local issues with transit networks to global wildlife preservation. However, regardless of scale, a critical bottleneck for many AI4SG initiatives is the laborious process of problem scoping—a complex and resource-intensive task—due to a scarcity of professionals with both technical and domain expertise. Given the remarkable applications of large language models (LLM), we propose a Problem Scoping Agent (PSA) that uses an LLM to generate comprehensive project proposals grounded in scientific literature and real-world knowledge. We demonstrate that our PSA framework generates proposals comparable to those written by experts through a blind review and AI evaluations. Finally, we document the challenges of real-world problem scoping and note several areas for future work. 1 University of Pittsburgh, 2 Carnegie Mellon University {jte27, ryanshi}@pitt.edu, [email protected] 1 Introduction AI for Social Good (AI4SG) has received considerable interest from the academia, industry, and government sectors ( Floridi et al. 2020 ; Shi, Wang, and Fang 2020 ) . This initiative uses AI methods to address societal issues, such as healthcare inequality ( Rawat, Joshi, and Kumar 2023 ; Zicari et al. 2021 ) , disparities in public transit ( Liu et al. 2020 ; Park et al. 2022 ) , and climate change ( Xu et al. 2020 ; Sisodia et al. 2023 ; Fang, Stone, and Tambe 2015 ) . Over the past decades, the effort, resources, and goodwill poured into this initiative have achieved great success, and also revealed many key bottlenecks. A key bottleneck is problem scoping, where the problem owners and stakeholders identify a pain point and how a particular AI method might solve it. Problem scoping can be a time-consuming and laborious process. It requires understanding the limitations of different technological methods and the constraints of the social issue at hand. Many public sector organizations lack the technical expertise to understand the former, while technologists are often negligent about the latter. When public sector organizations want to devote their already limited budget to technology projects, they have a hard time finding the relevant technical expertise, and end up with either missed opportunities or ill-scoped projects that cost them more down the road. Arguably one of the biggest successes of AI4SG initiatives is the emergence of researchers and practitioners who can bridge this gap. Yet they are too scarce in numbers compared to the abundant societal challenges and public sector organizations. If one could scale up this resource, even if at the cost of, say, only 60% as effective as experienced scoping professionals, one could greatly facilitate the promise and impact of AI4SG. We explore whether generative models can alleviate the resource requirements for problem scoping to promote future AI4SG initiatives. Large language models (LLMs) have been applied as mathematical provers ( Xin et al. 2024 ; Han et al. 2021 ; Jiang et al. 2022a ; Jiang et al. 2022b ) , research tools ( Lu et al. 2024 ; Pu et al. 2024 ; Baek et al. 2024 ; Si, Yang, and Hashimoto 2024 ) , and code generators ( Ifargan et al. 2024 ; Gu et al. 2024 ; Hu et al. 2024 ; Lu et al. 2024 ; Li et al. 2024 ) . Their inherent knowledge and ability to mimic human-like behaviors suggests they could automate or accelerate structured, information-intensive tasks. Problem scoping is one such task; it can be broken down into several general and repetitive steps requiring new information. Furthermore, unlike previous generations of AI methods, LLMs have been widely used by professionals who do not have technology expertise, including many in the public sector. As a result, it is much more natural for them to use LLMs to generate AI project ideas within their organizations. Admittedly, this affinity to LLMs is a double-edged sword. It is well-known that LLMs often fail to properly ground knowledge in a given domain, and could even generate false or inappropriate content. As we demonstrate later in the paper, using LLMs for scoping in a naive fashion would not be ideal. In this work, we use LLMs grounded by various information sources and queries to automate the scoping process. We make the following contributions. (1) We introduce the AI4SG problem scoping task, which has high real-world relevance and is also distinct to existing LLM tasks. (2) We propose the Problem Scoping Agent (PSA), an LLM pipeline enhanced by retrieval-augmented generation for coming up with AI4SG problem scopes. (3) We run experiments to compare our PSA with vanilla LLM methods and expert human professionals, and show that PSA achieves better scoping outcome than vanilla LLM methods and not too far from human experts in many cases. Figure 1: Problem Scoping Agent Overview 2 Related Work Idea Generation with AI Agents. Several prior studies evaluated the potential to accelerate research using generative models. Frameworks proposed in previous work generally take an iterative approach, where the current ideas are updated according to an outlined criteria and provided knowledge base. This knowledge base may be actively queried through an API or constructed before generating ideas. Iterative refinement ( Lu et al. 2024 ; Baek et al. 2024 ; Pu et al. 2024 ; Si, Yang, and Hashimoto 2024 ) , multimodule retrieval with feedback ( Yang et al. 2024 ) , and facet recombination ( Radensky et al. 2024 ) have shown success as fully automated systems, while other works focus on a collaborative procedure that improves brainstorming ( Lee, Hwang, and Lee 2024 ) . Similarly, several studies have tested whether generative models can operationalize or implement research projects by generating code and automating data analysis ( Ifargan et al. 2024 ; Gu et al. 2024 ; Hu et al. 2024 ; Lu et al. 2024 ; Li et al. 2024 ) . Distinct from these works, idea generation in a social-good context carries higher stakes and generally emphasizes less novel solutions in exchange for tractability and effectiveness. Most similar to our work is ( Zhao et al. 2024 ) where the authors outline the idea of foundation models as catalysts for collaborative social change. Differing from their work, we propose a fully-automated system and emphasize an exploration of problem scoping under a multiple contexts rather than resource allocation alone. LLMs as Automatic Evaluators. Several studies have additionally explored the use LLMs as evaluators and reviewers. While LLMs demonstrate a non-trivial performance on pair-wise comparisons ( Si, Yang, and Hashimoto 2024 ) , it is unclear whether such models can empirically evaluate solutions. Some work suggests that LLM achieves a higher agreement with human reviewers than reviewers themselves ( Lu et al. 2024 ) ; however, other studies find that the agreement between human reviewers and LLMs is less than random ( Si, Yang, and Hashimoto 2024 ; Stureborg, Alikaniotis, and Suhara 2024 ) . Furthermore, LLMs demonstrate a preference for generations with a lower perplexity, typically their own generations, over human-written text ( Panickssery, Bowman, and Feng 2024 ) . Additional studies report that LLMs

Read the complete text