Audit, Alignment, and Optimization of LM-Powered Subroutines with Application to Public Comment Processing

Reilly Raab, Mike Parker, Dan Nally, Sadie Montgomery, Anastasia Bernat, Sai Munikoti, Sameera Horawalavithana · 2025

Read Audit, Alignment, and Optimization of LM-Powered Subroutines with Application to Public Comment Processing by Reilly Raab, Mike Parker, Dan Nally, Sadie Montgomery, Anastasia Bernat, Sai Munikoti, Sameera Horawalavithana (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-08109.txt.

Introduction

Reilly Raab and co-authors wrap language models in auditable software routines in a 2025 paper. Agencies must process floods of public comments on projects needing environmental review. Models could help, but officials worry about bias, safety, and hidden workings. The team's framework declares typed, function-like routines powered by language models, callable from ordinary code.

Every prompt, input, output, and link between them is logged for later audit. Sparse fixes from human experts improve each routine while it runs, so experts judge rather than wrangle prompts. Built on the framework, their CommentNEPA tool sorts and summarizes comment sets for impact statements. Scored against human-labeled historical ground truth, the unattended system performed credibly. The authors offer the library for other high-stakes text work.

Catalog record

CREATORReilly Raab, Mike Parker, Dan Nally, Sadie Montgomery, Anastasia Bernat, Sai Munikoti, Sameera Horawalavithana
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE53 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2507-08109

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Audit, Alignment, and Optimization of LM-Powered Subroutines with Application to Public Comment Processing Reilly Raab, Mike Parker, Dan Nally, Sadie Montgomery, Anastasia Bernat, Sai Munikoti, Sameera Horawalavithana Abstract The advent of language models (LMs) has the potential to dramatically accelerate tasks that may be cast to text-processing; however, real-world adoption is hindered by concerns regarding safety, explainability, and bias. How can we responsibly leverage LMs in a transparent, auditable manner—minimizing risk and allowing human experts to focus on informed decision-making rather than data-processing or prompt engineering? In this work, we propose a framework for declaring statically typed, LM-powered subroutines (i.e., callable, function-like procedures) for use within conventional asynchronous code—such that sparse feedback from human experts is used to improve the performance of each subroutine online (i.e., during use). In our implementation, all LM-produced artifacts (i.e., prompts, inputs, outputs, and data-dependencies) are recorded and exposed to audit on demand. We package this framework as a library to support its adoption and continued development. While this framework may be applicable across several real-world decision workflows (e.g., in healthcare and legal fields), we evaluate it in the context of public comment processing as mandated by the 1969 National Environmental Protection Act (NEPA): Specifically, we use this framework to develop “CommentNEPA,” an application that compiles, organizes, and summarizes a corpus of public commentary submitted in response to a project requiring environmental review. We quantitatively evaluate the application by comparing its outputs (when operating without human feedback) to historical “ground-truth” data as labelled by human annotators during the preparation of official environmental impact statements. 1 Introduction Contemporary organizations have shown great interest in integrating language models (LMs) into workflows traditionally performed by human subject matter experts (SMEs), such as in medical diagnostics ( Artsi et al. 2025 ) , legal assistance ( Padiu et al. 2024 ) , financial risk analysis ( AI21 labs 2025 ) , and governmental permitting or regulatory reviews ( Phan et al. 2024 ) . Despite this interest, however, the use of LMs (e.g., via a standard conversational interface) in high-stakes contexts is constrained by the need for decision-making reliability, objectivity, transparency, and accountability that SMEs currently provide ( Mori 2024 ) . Effective reconciliation between LMs and SMEs thus represents a critical frontier in real-world deployments of artificial intelligence. LMs have demonstrated remarkable capabilities in extracting information from large volumes of multi-modal, multi-domain data; synthesizing multi-document concepts; and performing tasks associated with basic reasoning. Nonetheless, LMs are susceptible to “hallucinations” (i.e., inaccurate generation) ( Ji et al. 2023 ) , difficulty in handling nuanced, domain-specific requirements ( Ashqar 2025 ) , historical biases inherited from training data ( Ranjan et al. 2024 ) , and opaque reasoning in decision-making ( Machot et al. 2024 ) . Notably, these weaknesses are often precisely the strengths of SMEs, who are conversely burdened with the inefficient and labor-intensive tasks of cross-document, multi-modal search and information extraction. We can see the need to delineate and integrate the often low-stakes or tedious work that can be performed by LMs with the discerning, high-stakes decision-making tasks performed by SMEs in the real world: The challenge is to harness the time efficiency and broad knowledge capabilities of LMs while preserving the domain expertise, contextual judgment, oversight, and accountability of SMEs. Moreover, we must do so without creating additional burdens for SMEs to work with LMs (e.g., “prompt-engineering” or manual review of all LM tasks), and we wish to minimize the introduction of new risks (e.g., a loss of clarity regarding where or how LMs may be used by each SME, or, in the case of governmental work, the erosion of public trust). In this work, we propose a novel auditable and interactive refinement framework for the effective integration of LMs with SMEs for decision-making workflows. In this framework, LMs are framed as execution engines within the well-scoped, text-based subroutines 1 1 1 In this work, our use of “subroutines” refers to named blocks of code that can be invoked with inputs of pre-specified type to produce outputs of pre-specified type, though the outputs may depend on internal state or randomness. , which are auditable by SMEs. This formulation further accommodates sparse feedback provided by the SMEs to optimize the performance of individual subroutines, aligning work performed by the LM with aggregate SME expectations and preferences over time. We demonstrate the applicability and effectiveness of this proposed framework in the context of the U.S. federal National Environmental Protection Act of 1969 (NEPA) permitting review process, particularly focusing on streamlining the public comment processing workflow. We selected the NEPA public comment processing as the test case due to its pressing need and significant social and environmental impact. Currently, the public comment processing process is highly tedious and manual, resulting in substantial economic burdens and project delays for federal agencies. Our framework aims to address some of these challenges, thereby supporting the core objective of the NEPA (i.e., effective communication of the environmental impact of the proposed projects to benefit and inform the public). The need for AI integration in public comment processing is further reinforced by the growing interest of the U.S. federal government in streamlining federal permitting processes ( The White House 2025 ) . Primary Contributions The contributions of this work are as follows: 1. A methodology for optimizing subroutine prompts combining a sampling method targeting infinitely many-armed bandit ( \infty -MAB) regret minimization and self-critique loops to leverage sparse human feedback. 2. An open-source software library 2 2 2 A link to the library will be included pending internal disclosure processes. that implements this contributed methodology, suitable for static-type analysis in Python and integrated with a relational database for tracking LM outputs and user feedback for online audit. 3. Initial evaluation of an application , called CommentNEPA, built with the contributed library, specialized for public comment processing. In this work, we develop a framework for the auditable use of LMs by SMEs, capable of dramatically accelerating the processing of public comments related to environmental permitting under NEPA. This technical framework, which we detail in Section 2 , benefits from sparse feedback provided by the SMEs to optimize the performance of individual subroutines, aligning work performed by the LM with aggregate SME expectations and preferences. We compare the outputs achieved through the use of this framework, through an application called “CommentNEPA,” detailed in Section 3 , to historical outputs achieved from SME labor ( Section 4 ). 1.1 Application Domain: Comment Processing for Environmental Review Public participation in environmental review under NEPA plays a critical and statistically significant role in shaping decisions and outcomes in the United States ( Stava et al. 2025 ) . Unfortunately, comment processing for environmental review is tedious and labor-intensive for SMEs, resulting in delays that are a frequent target of political criticism. Despite pressure for accelerated processing, demands on expert labor is further exacerbated by an increasing volume of projects and public comments, which now also leverage generative AI. As we consider the use of generative AI to help SMEs accelerate comment processing while maintaining and supporting public engagement, we must acknowledge the sensitivity of this topic: Inappropriate use of AI may introduce risks of litigation, a loss of public

Read the complete text