Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design

Marco Wrzalik, Adrian Ulges, Anne Uersfeld, Florian Faust, Viola Campos · 2024

Read Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design by Marco Wrzalik, Adrian Ulges, Anne Uersfeld, Florian Faust, Viola Campos (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-06432.txt.

Introduction

Marco Wrzalik and co-authors compare two ways to teach language models with expert labels in this 2024 paper. Their task is to spot emission cut goals in company reports, a step in tracking firm climate pledges. Expert-marked sample passages can guide a model either as live examples or as fuel for rewriting its task prompt.

Tests on a public set of close to eight hundred passages from real reports favor the prompt route. Letting the model tune its own prompt beat picking fresh examples for each passage. Mixing both routes added little. A close read shows the tuned prompts catch many fine points of the task. The result guides teams that build such checks on a budget.

Catalog record

CREATORMarco Wrzalik, Adrian Ulges, Anne Uersfeld, Florian Faust, Viola Campos
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE26 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-06432

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Integrating Expert Labels into LLM-based Emission Goal Detection: Example Selection vs Automatic Prompt Design Marco Wrzalik, Adrian Ulges, Anne Uersfeld, Florian Faust, Viola Campos Abstract We address the detection of emission reduction goals in corporate reports, an important task for monitoring companies’ progress in addressing climate change. Specifically, we focus on the issue of integrating expert feedback in the form of labeled example passages into LLM-based pipelines, and compare the two strategies of (1) a dynamic selection of few-shot examples and (2) the automatic optimization of the prompt by the LLM itself. Our findings on a public dataset of 769 climate-related passages from real-world business reports indicate that automatic prompt optimization is the superior approach, while combining both methods provides only limited benefit. Qualitative results indicate that optimized prompts do indeed capture many intricacies of the targeted emission goal extraction task. 1 Introduction The urgency of the climate crisis necessitates immediate action across various sectors of the global economy. These efforts are targeted at Net Zero , i.e. achieving a balance between emitted and removed greenhouse gases, particularly CO 2 . Policies such as the European Union’s Green Taxonomy aim to redirect financial investment flows toward sustainable businesses, setting incentives for companies committing to concrete emission reduction goals. To evaluate and monitor these commitments, analysts must review extensive corporate documentation, including annual reports, sustainability reports, and stewardship disclosures. This manual process of locating and extracting relevant data, often referred to as carbon accounting , is labor-intensive and time-consuming. Identifying such climate goals is a surprisingly intricate task, as analysts have to distinguish concrete, binding, self-imposed and quantitative emission goals from vague statements, greenwashing, externally mandated requirements, goals related to climate change (such as "moving out of coal"), etc. Overall, the task not only requires high precision but also poses unique challenges due to the nuanced language and diverse reporting styles used by companies. To increase the efficiency of carbon accounting, Large Language Models (LLMs) have emerged as powerful tools. Given a prompt consisting of task-specific instructions and a text passage to analyze, the LLM outputs whether the passage contains an emission goal. When integrating such LLMs into practical workflows, their outputs are inspected and – in case of errors – corrected by analysts, resulting in a set of labeled "challenge samples" that grows over time. These examples offer an interesting option for in-context learning (ICL), i.e. to improve the LLM’s accuracy by utilizing the examples to improve the prompt, without applying fine-tuning to the model. In this paper, we investigate ICL strategies to improve LLMs with expert knowledge in form of labeled examples, focusing on the task of emission goal detection. Particularly, we compare two approaches: (1) Example selection , which incorporates a limited number of few-shot examples into the prompt. These exemplify the desired behavior, and are selected dynamically to resemble the input passage. (2) Automatic Prompt Design , in which the LLM adjusts its own instructions. This approach applies an iterative optimization process in which erroneous challenge cases are inspected, reflected, and new, refined instructions are generated and evaluated. While both techniques hold promise, their effectiveness in real-world applications has not been compared extensively. We hope to fill this gap with the following contributions: 1. We conduct a comprehensive comparison of example selection and automatic prompt design on the task of emission goal classification, using 769 passages from the public NetZeroFacts dataset Wrzalik et al. (2024) . 2. We highlight the strengths and limitations of each prompting strategy, guiding practitioners applying LLMs to sustainability classification tasks. Specifically: (1) example selection improves results, especially with weak prompts; (2) auto-prompting yields greater overall gains; and (3) in automatic prompt design, few-shot examples provide limited additional benefits. 2 Related Work This section reviews existing literature on prompting strategies that make use of labeled data, focusing on example selection for few-shot learning and automatic prompt design. 2.1 Example Selection for Few-Shot Learning Liu et al. (2022) investigated the selection of good in-context examples for GPT-3 and found that semantically similar examples, chosen based on proximity in embedding space, significantly improve model performance. Rubin et al. (2022) proposed a contrastive learning-based method that learns to retrieve task-specific examples, showing significant performance improvements by optimizing the input-output pairs for correct predictions. Su et al. (2023) introduced a selective annotation framework that enhances few-shot learning by strategically selecting diverse and representative examples from a small annotated pool. Their graph-based approach demonstrates that such careful example selection can lead to significant performance improvements across various natural language processing tasks. Zhang et al. (2022) framed example selection as a reinforcement learning problem, proposing a Q-learning-based approach to actively select examples. This method shifts from similarity-based retrieval to learning a policy that optimizes example selection for few-shot learning, demonstrating moderate improvements on downstream tasks. In this work, we focus on the selection of semantically similar examples as motivated by Liu et al. (2022) . 2.2 Automatic Prompt Design The task of optimizing prompts for LLMs has received increasing attention, with several approaches leveraging the model’s ability to self-improve: Shin et al. (2020) introduced AutoPrompt uses a gradient-guided search to iteratively refine trigger tokens, resulting in competitive performance compared to manually designed prompts for tasks like natural language inference. Zhou et al. (2023) proposed Automatic Prompt Engineer (APE), which iterates over LLM-generated candidate prompts, selecting and refining those that lead to the best performance based on task-specific score functions, often outperforming human-generated prompts. Similarly, Pryzant et al. (2023) introduced ProTeGi , a method that optimizes prompts based on LLM-generated rasonings over erroneous examples. The performance of these reasonings is estimated and stears a beam search that explores multiple candidate options. Yang et al. (2024) utilize the LLM itself as a general-purpose optimizer capable of refining prompts and solving various tasks. Their method, OPRO, iteratively generates and evaluates prompts based on the history of previous attempts, framing prompt optimization as a general meta-optimization task. Finally, Intent-based Prompt Calibration (IPC) by Levi et al. (2024) introduces the auto-generation of synthetic challenge cases to calibrate prompts, which are then labeled by the expert. Our work explores automatic prompt design – as outlined above – from a practitioners perspective. Specifically, we follow a simplified variant of Pryzant et al. (2023) ’s ProTeGi , which (instead of beam search) performs a greedy search. Table 1: Results of Few-shot Prompting (%, including example selection in Row 4 ("Similar")). simple instructions expert instructions Examples Acc Prec Rec F1 Acc Prec Rec F1 Zero-shot 56.2 52.2 100.0 68.6 81.2 77.1 86.1 81.4 Static 72.3 63.8 97.0 77.0 82.0 90.8 69.4 78.7 Random 66.2 58.9 98.6 73.7 80.5 75.7 89.5 81.5 Similar 69.8 62.3 92.9 74.6 87.0 87.2 85.4 86.3 2.3 Emission Information Extraction The detection and extraction of SDG (Sustainable Development Goals) related information has recently gained traction as a research field: Spokoyny et al. (2023) bridge the gap between structured reporting (in form of questionnaires) and unstructured reporting in form of text: Their ClimaBench benchmark challenges NLP models to extract climate-related information from company reports and thus auto-fill questionnaires. Schimanski et al. (2024) pretrain specific BERT models for environmental, social and governance aspects on 13.8 million corporate disclosures and curate three balanced 2k - document test

Read the complete text