SusGen-GPT: A Data-Centric LLM for Financial NLP and Sustainability Report Generation
Qilong Wu, Xiaoneng Xiang, Hejia Huang, Xuan Wang, Yeo Wei Jie, Ranjan Satapathy, Ricardo Shirota Filho, Bharadwaj Veeravalli · 2024
Read SusGen-GPT: A Data-Centric LLM for Financial NLP and Sustainability Report Generation by Qilong Wu, Xiaoneng Xiang, Hejia Huang, Xuan Wang, Yeo Wei Jie, Ranjan Satapathy, Ricardo Shirota Filho, Bharadwaj Veeravalli (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10906.txt.
Introduction
Qilong Wu leads a team building an open language model for finance and green reporting. Their paper appeared in two thousand twenty-four. Banks and firms face growing demands around climate and social reporting. Open models that handle both finance tasks and such reports are rare. The authors set out to supply that missing tool.
They built SusGen-30K, a balanced dataset with seven finance tasks plus report writing. They also made TCFD-Bench to grade sustainability report generation. Their SusGen-GPT models top most of the tested tasks. The models trail a far larger closed model by a small margin. A full system adds retrieval support to help draft reports. The work targets firms that must write clear, rule-based disclosures.
Catalog record
| CREATOR | Qilong Wu, Xiaoneng Xiang, Hejia Huang, Xuan Wang, Yeo Wei Jie, Ranjan Satapathy, Ricardo Shirota Filho, Bharadwaj Veeravalli |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 64 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10906
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
SusGen-GPT: A Data-Centric LLM for Financial NLP and Sustainability Report Generation Qilong Wu, Xiaoneng Xiang, Hejia Huang, Xuan Wang, Yeo Wei Jie, Ranjan Satapathy, Ricardo Shirota Filho, Bharadwaj Veeravalli Abstract The rapid growth of the financial sector and the increasing focus on Environmental, Social, and Governance (ESG) considerations have created a pressing need for advanced natural language processing (NLP) tools. Despite recent advancements, there is still a notable absence of open-source Large Language Models (LLMs) that are proficient across both general finance and ESG domains, such as generating ESG reports. To address this gap, we introduce SusGen-30K , a high-quality, category-balanced dataset comprising seven financial NLP tasks and ESG report generation. In addition, we propose TCFD-Bench , a benchmark designed to improve the evaluation of sustainability report generation. Our data-centric approach led to the development of a suite of models, SusGen-GPT , trained on the curated dataset. These models were evaluated across six adapted tasks and two off-the-shelf tasks, showing state-of-the-art performance, surpassing all other models except GPT-4. Remarkably, SusGen-GPT achieved an average score only 0.02 below GPT-4, despite using models with only 7-8B parameters compared to GPT-4’s 1,700B. This demonstrates the efficiency of our approach in delivering high performance with significantly fewer resources, addressing existing challenges and fostering further advancements in the financial and ESG research community. 1 Introduction As automation in the financial sector gains more attention and climate change becomes an increasingly urgent issue, there is a growing need for more advanced tools capable of solving these problems. The ability to efficiently process and generate financial reports, analyze ESG metrics, and produce comprehensive TCFD-format 1 1 1 https://www.fsb-tcfd.org/ . We investigated various sustainability reporting guidelines, including GRI, SASB, EU CSRD, etc., and ultimately chose TCFD because it is more standardized and universally applicable. reports is crucial for maintaining transparency and accountability. Recently, LLMs ( Brown et al., 2020 ; Ouyang et al., 2022 ; Touvron and et. al., 2023a ; OpenAI, 2023a ; Touvron and et. al., 2023b ) have gained widespread popularity due to their ability to solve complex tasks such as commonsense reasoning, machine translation or even performing self-training ( Yeo et al., 2024 ) . However, there remains a substantial gap in LLMs tailored towards specialized domains such as finance and ESG ( Liu et al., 2023 ; Wang et al., 2023a ) . This is likely due to the training dataset being exposed to mostly general text documents and lack data of specialized information. (a) Performance comparison (Relative) with baseline models on each financial NLP task with average score. (b) Overall scores of financial NLP benchmark across 6 tasks. Dashed lines for two LLM types’ average. Figure 1: An overview of model comparison with both open-source and closed-base baseline models on six financial NLP tasks. The two sub-figures show that our model SusGen-GPT achieves state-of-the-art performance in most benchmarks. In light of these challenges, we propose SusGen-30K, a meticulously curated dataset that is designed to tackle multiple NLP tasks across both financial and ESG domains. More importantly, we introduce a suite of LLMs, trained on our proposed dataset, which we refer to as SusGen-GPT. SusGen-GPT is capable of achieving superior performance across multiple downstream tasks simultaneously, when compared against models that are orders of magnitude larger. Additionally, we propose a new benchmark, TCFD-Bench, specifically designed to assess models’ ability to generate concise and accurate ESG reports from annual reports. We likewise conduct experiments on the proposed benchmark using SusGen-GPT. In total, our contributions include the following: 1. SusGen-30K, a large-scale high quality dataset in both financial & ESG domain. 2. We propose and release a well-curated benchmark, tailored towards ESG report generation, TCFD-Bench. 3. A suite of fine-tuned LLMs, SusGen-GPT, shown to achieve comparable state-of-the-art performance to GPT-4 across both general financial and ESG NLP benchmarks as shown in Figure 1 when most of open-source models struggle to perform well in these domains. Remarkably, our models only have only 7-8B parameter models even with quantization comparing to the GPT4’s 1,700B parameters or so, making it a computational efficient solution. 2 Related Work and Background NLP for Finance & ESG Natural Language Processing (NLP) has found extensive applications in various financial tasks, demonstrating its versatility and depth in addressing diverse financial issues ( Masson and Paroubek, 2024 ; Aguda et al., 2024 ) . The key tasks in the financial domain include Question Answering (QA), Headline Classification (HC), and report generation. More notably, there exists a gap in achieving an acceptable level of proficiency in automating the generation of ESG reports. One such effort, ChatReport ( Ni et al., 2023 ) is developed to perform summarization and analysis on ESG reports. However, these tools face challenges such as generating reports that are overly simplified and lacking important details. Other attempts ( Bronzini et al., 2024 ; Zou et al., 2023 ; Luccioni et al., 2020 ) mainly rely on existing data extraction techniques and face difficulty in processing unstructured data. Our dataset aims to bridge these gaps by providing a data-centric approach to training LLMs in a multi-task manner. General Large Language Models Given the increased accessibility to large amounts of publicly available data, there has been a constant upward trend in releasing instruct-tuned LLMs. These models include Alpaca ( Taori et al., 2023 ) , an LLM trained on a dataset augmented with GPT-3. Recently, the latest open-source LLMs, Mistral-v0.3 ( Jiang et al., 2023 ) and Llama3 ( Dubey and et. al, 2024 ) , have joined the community, showcasing impressive human-like capabilities across various domains. However, these models are not tailored to any specific domain and often underperform in specialized areas such as finance and ESG. Financial Large Language Models Financial Large Language Models (FinLLMs) are specifically developed to handle financial text data, offering more precise financial analysis and predictions. One of the earlier efforts, BloombergGPT ( Wu et al., 2023 ) , is a 50B model trained on a massive dataset comprising a mixture of financial and general text data. However, it is not publicly accessible and hence there is a call for more open and inclusive alternatives. Other open-source alternatives includes FinGPT ( Yang et al., 2023 ; Liu et al., 2023 ) and CFGPT ( Lei et al., 2024 ) , which introduce tools focused on data acquisition, cleaning, and preprocessing. Their goal is to democratize financial data and the development of FinLLMs, offering a wide range of potential applications. Nonetheless, these efforts have not addressed key concerns on the imbalance in training data and lack of knowledge in the ESG domain. CFGPT faces limitations in language such as only being limited to the Chinese language. Financial Benchmarks As FinLLMs rapidly advance, the importance of financial evaluation benchmarks has grown significantly. For example, FinGPT Benchmarks and FLUE, focused on assessing NLP tools on a wide array of tasks such as NER and SA. PIXIU ( Xie and et. al., 2023 ; Xie and et. al., 2024 ) is a large-scale multitask dataset containing 136K data samples as well as offering benchmarks covering five downstream tasks. However, these evaluation frameworks lack specialized ESG content. We aimed to bridge this gap by introducing TCFD-Bench. 3 SusGen-GPT 3.1 Framework The system,