Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysis

Bo Hu & Han Yuan & Vlad Pandelea & Wuqiong Luo & Yingzhu Zhao & Zheng MaGlobal Decision Science, American Express {Bo.Hu, Han.Yuan1, Vlad.A.Pandelea, Wuqiong.Luo,Yingzhu.Zhao, · 2025

Read Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysis by Bo Hu & Han Yuan & Vlad Pandelea & Wuqiong Luo & Yingzhu Zhao & Zheng MaGlobal Decision Science, American Express {Bo.Hu, Han.Yuan1, Vlad.A.Pandelea, Wuqiong.Luo,Yingzhu.Zhao, (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-16575.txt.

Introduction

Bo Hu and co-authors set out a new way to grade long language model answers in finance work. Their paper from 2025 starts from a gap: standard scores work for short texts but fade on long ones. Real tasks like finance review and rule checks join a long question, a long context, and a long answer.

The team built a real finance data set of such triplets and showed where old scores fail. Their method, called Extract, Match, and Score, pulls key parts out, pairs them, and grades the pairs. It is shaped for the knots of long form output. The authors offer it as a sound grading path for hard real world model use.

Catalog record

CREATORBo Hu & Han Yuan & Vlad Pandelea & Wuqiong Luo & Yingzhu Zhao & Zheng MaGlobal Decision Science, American Express {Bo.Hu, Han.Yuan1, Vlad.A.Pandelea, Wuqiong.Luo,Yingzhu.Zhao,
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE39 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2503-16575

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Extract, Match, and Score: An Evaluation Paradigm for Long Question-context-answer Triplets in Financial Analysis Bo Hu & Han Yuan & Vlad Pandelea & Wuqiong Luo & Yingzhu Zhao & Zheng MaGlobal Decision Science, American Express {Bo.Hu, Han.Yuan1, Vlad.A.Pandelea, Wuqiong.Luo,Yingzhu.Zhao, Abstract The rapid advancement of large language models (LLMs) has sparked widespread adoption across diverse applications, making robust evaluation frameworks crucial for assessing their performance. While conventional evaluation metrics remain applicable for shorter texts, their efficacy diminishes when evaluating the quality of long-form answers. This limitation is particularly critical in real-world scenarios involving extended questions, extensive context, and long-form answers, such as financial analysis or regulatory compliance. In this paper, we use a practical financial use case to illustrate applications that handle “long question-context-answer triplets”. We construct a real-world financial dataset comprising long triplets and demonstrate the inadequacies of traditional metrics. To address this, we propose an effective Extract, Match, and Score (EMS) evaluation approach tailored to the complexities of long-form LLMs’ outputs, providing practitioners with a reliable methodology for assessing LLMs’ performance in complex real-world scenarios. † † footnotetext: † These authors contributed equally to this work. † † footnotetext: 🖂 Correspondence: Zheng Ma, Singapore Decision Science Center of Excellence, American Express, 1 Marina Boulevard, 018989, Singapore. 1 Introduction The advancement of large language models (LLMs) has revolutionized automated model-based solutions to an unprecedentedly accuracy that rivals human performance across a series of natural language processing (NLP) tasks ( Yang et al., 2024 ) . Despite their impressive performance on short-form tasks ( Chang et al., 2024 ) , the applicability and effectiveness of LLMs in long-form scenarios warrant in-depth exploration, particularly those where short-form summaries lead to substantial information loss ( Shaham et al., 2023 ; Wijesiriwardene et al., 2023 ) . Preliminary studies have investigated long-form challenges in the general domain using either standalone LLMs ( Han et al., 2023 ; Liu et al., 2024 ; Li et al., 2024 ; Wang et al., 2024 ; Laban et al., 2024 ) or retrieval-augmented generation (RAG) systems ( Su et al., 2022 ; Yu et al., 2023 ; Zhao et al., 2024a ; Qi et al., 2024 ; Han et al., 2024 ; Laban et al., 2024 ) . Compared to the general domain, long-form scenarios are particularly prevalent in the financial sector, where documents such as financial statements and compliance reports often exceed hundreds of pages ( Liu et al., 2022 ; Reddy et al., 2024 ; Zhao et al., 2024b ) . However, the absence of effective evaluation metrics poses a significant challenge in financial NLP. In particular, traditional metrics lose their efficacy as evaluation tasks increase in length and complexity. Consequently, many studies in financial NLP simplify complex, open-ended analytical tasks into well-defined, closed-ended tasks such as news sentiment analysis ( Guo et al., 2023 ; Chen et al., 2024b ; Kirtac & Germano, 2024 ; Shah et al., 2023 ) , concept classification ( Fonseca & Cohen, 2024 ) , name entity recognition ( Shah et al., 2022 ; Guo et al., 2023 ; Bhatia et al., 2024 ) , structure boundary detection ( Shah et al., 2022 ) , event extraction ( Zhou et al., 2021 ) , relation identification ( Li et al., 2023 ) , etc. While some datasets focus on open-ended tasks such as question answering ( Shah et al., 2022 ; Reddy et al., 2024 ; Krumdick et al., 2024 ; Chen et al., 2024a ) and abstractive summary ( Zhu et al., 2024 ) , they typically address only one or two of the three key aspects that possess the long-form property: context ( Reddy et al., 2024 ; Srivastava et al., 2024 ; Zhu et al., 2024 ; Xie et al., 2023 ; Zhao et al., 2024b ; Gupta et al., 2024 ) , question ( Krumdick et al., 2024 ; Chen et al., 2022 ) , and answer ( Band et al., 2024 ; Qi et al., 2024 ) . To the best of our knowledge, while a few efforts have been made in the general domain ( Tan et al., 2024 ) , no existing work directly addresses the evaluation challenge of long triplets in finance. To systematically investigate the current landscape and advance the research on financial long triplets, we (1) Benchmarked the state-of-the-art LLMs on long-form analysis of earning call transcripts from 10 largest constituents in S&P 500 index; (2) Identified the inefficiency of conventional evaluation methods, such as ROUGE, in differentiating generation quality within the context of long triplets ( Xu et al., 2023 ; Tan et al., 2024 ) ; (3) Proposed a generalizable evaluation paradigm named EMS ( E xtract, M atch, and S core) to provide fine-grained evaluations and demonstrated its competitive advantages over RAGChecker ( Ru et al., 2024 ) in real-world scenarios of long triplets. 2 Methodology To address the inefficiency of conventional evaluation approaches in the context of long triplets, we propose a three-stage methodology EMS : (1) Extracts salient points from both the reference and candidate answers; (2) Identifies potential alignments between these points; (3) Scores each matched pair based on predefined criteria. The long-form generated answer is denoted as ans , and the corresponding reference is denoted as ref . Our evaluation pipeline takes ans and ref as input, and provides three saliency points level metrics: EMS-Recall, EMS-Precision, EMS-F1. 2.1 Stage 1: Saliency Point Extraction Given a long-form reference answer, we define a function f^{(\text{E})}(\cdot) that extracts a list of saliency points from the text. Formally, P^{(\text{ref})}=f^{(\text{E})}(ref)=[p_{1}^{(\text{ref})},\dots,p_{N}^{(\text{ref})}], (1) where P^{(\text{ref})} denotes the set of saliency points extracted from the reference. We apply the same function f(\cdot) to the candidate answer: P^{(\text{ans})}=f^{(\text{E})}(ans)=[p_{1}^{(\text{ans})},\dots,p_{M}^{(\text{ans})}], (2) yielding the set P^{(\text{ans})} , which contains the saliency points extracted from the candidate answer. The purpose of saliency points extraction is to decompose the long-form answer into key claims, ensuring that the evaluation score of these claims accurately reflects the quality of the original response. To achieve this, the saliency point extractor should follow well-crafted instructions. Figure 1: An example of saliency point extraction from long-form answer. Including details. Saliency points are not brief summaries from the original answer, since the detailed information is useful to users and shows as a sign of good quality. Hence, saliency points should maintain all the details from the original long-form answer, so that the scoring over saliency points can represent the quality of original answers. Figure 1 shows an example that the detailed numbers related to the revenue is critical to users with finance background. The shorter extracted point is factually accurate, however, it filters out indispensable information. Retaining repeated points. Low-quality answers are not solely the result of missing or incorrect claims, which can be penalized during scoring, but also stem from poor organization of those claims. Specifically, a long-form answer should be assigned with a low score if it contains repeated content, even when the saliency points from those repetitions are accurate. Therefore, all repeated points are retained during extraction, leading to a saliency point list that includes all duplicates. Removing summary. An exception to the treatment of repeated content is with the summaries from the long-form answers. High-quality long-form answers typically include

Read the complete text