When Sufficient is not Enough: Utilizing the Rashomon Effect for Complete Evidence Extraction

Katharina Beckh, Stefan Rüping · 2025

Read When Sufficient is not Enough: Utilizing the Rashomon Effect for Complete Evidence Extraction by Katharina Beckh, Stefan Rüping (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-07055.txt.

Introduction

Katharina Beckh, Sven Heuser, and Stefan Rüping formulate a task they call complete evidence extraction. Decision support systems often justify an output with a short sufficient snippet, but regulatory compliance and medical billing demand all the input tokens that support a decision, not just enough of them. Working in medical coding, the authors ask how fully automated systems can recover that full evidence.

Their approach exploits the Rashomon effect: many other models perform equally well while attending to different evidence. Aggregating token-level attributions from an ensemble of language models therefore recovers evidence any single model misses. On a dataset with human-labeled evidence, ensembles of only three models already beat the best single model, greatly raising evidence recall for only a small extra token overhead.

Catalog record

CREATORKatharina Beckh, Stefan Rüping
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE18 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-07055

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

When Sufficient is not Enough: Utilizing the Rashomon Effect for Complete Evidence Extraction Katharina Beckh, Stefan Rüping Abstract Feature attribution methods typically provide minimal sufficient evidence justifying a model decision. However, in many applications this is inadequate. For compliance and cataloging, the full set of contributing features must be identified – complete evidence. We perform a case study on a medical dataset which contains human-annotated complete evidence. We show that individual models typically recover only subsets of complete evidence and that aggregating evidence from several models improves evidence recall from ∼ \sim 0.60 (single best model) to ∼ \sim 0.86 (ensemble). We analyze the recall-precision trade-off, the role of training with evidence, dynamic ensembles with certainty thresholds, and discuss implications. 1 Introduction Evidence extraction identifies input features that justify a model prediction. In most prior work, the focus is on minimal sufficient justifications 10 ; 18 ; 11 . However, in certain settings, such as regulatory compliance or billing, stakeholders need complete evidence which covers all supporting text spans. This especially applies to scenarios with long texts, where relevant information is distributed across the document. Completeness is essential for compliance and cataloging. For example, standards and specifications may undergo revisions when new technologies emerge which are not captured yet by the rules in the existing standard document 16 ; 1 . It is therefore necessary to identify all information to be modified. This also applies to compliance scenarios where texts must be revised to ensure conformity with new regulations. Moreover, completeness matters when the amount of evidence influences decision-making. For instance, in psychiatric care, the number and severity of indicators, such as self-harm and aggression, determine the need for hospitalization and impact billing 13 . Figure 1: Illustration of complete evidence extraction on the medical coding task. Model 1 identifies two relevant evidence spans in the input document, Model 2 one. Aggregating the evidence of several models leads to higher recall and, thus, more complete evidence. Achieving completeness is challenging in practice because its goal diverges from the standard classification objective. For the latter, the goal is to find the determining factors to distinguish between different classes. Models latch on a set of discriminative features, recovering only a subset of relevant evidence. Hence, a single model will not suffice to provide complete evidence. The Rashomon effect implies that different models achieve similar performance while relying on distinct solution strategies 3 ; 15 ; 12 . We use the effect to our advantage: if different models attend to different valid cues, then aggregating their evidence may increase coverage. In this work, we empirically investigate complete evidence extraction using a medical dataset with human-annotated evidence (Fig. 1 ). We compare the evidence of single models to ensembles formed by aggregating evidence from multiple models. We examine the trade-off between recall and precision, the role of evidence-guided training, and whether limiting ensemble members by certainty helps in reducing precision loss. 2 Sufficient and Complete Evidence Here, evidence refers to input tokens that support a model prediction. Evidence does not necessarily provide insight into how a prediction was reached 17 which is why we use the more specific term evidence instead of explanation. We adopt the distinction by 4 : sufficient evidence is any subset of text that, on its own, justifies a model prediction, whereas complete evidence is the exhaustive set of all text in the document that supports that prediction. While the term completeness suggests a binary nature, we view completeness as a spectrum as to how much evidence is covered. Most related work, especially in the field of explainable AI, focuses on sufficiency 10 ; 11 ; 18 . While the notion of comprehensiveness 5 is similar, completeness does not assume that evidence removal leads to a reduction in model confidence. 3 Case Study: Evidence Extraction in Medical Coding Due to the Rashomon effect, we expect that a single model only covers parts of all evidence. In a case study, we confirm this phenomenon and empirically show that aggregating evidence from several models leads to more complete evidence. We use a dataset from the medical domain which contains complete evidence annotated by human experts. The setting is a classification task and evidence is extracted via a feature attribution method. Completeness is seen as the set of all tokens in the input that justify the label and we use recall as the main evaluation measure. We compare recall values of single models and ensembles. It is explored whether using evidence only from models with high certainty leads to similar or higher recall. To assess the impact of different training regimes, a comparison between regular and evidence-guided training is performed. 3.1 Task and Dataset We consider the classification task of assigning medical codes to free-text clinical notes as the basis task and focus our analysis on the evidence extraction task, realized by retrieving text snippets that support a specific code. MDACE 4 is a medical dataset based on MIMIC-III 7 ; 8 . It contains electronic health records in the English language with diagnosis and procedure codes. Here, ICD-9 codes are utilized. For each code, evidence is provided in the form of annotated text spans in the clinical note. For example, code 416.8 (other pulmonary heart disease) has ‘pulmonary hypertension’ as one evidence span. MDACE contains documents which are annotated in a sufficient style, and also a complete style, annotating all text that is relevant for a code. The complete subset has roughly three times more evidence compared to the sufficient subset. Following established set-ups, discharge summaries serve as data basis and the sufficient evidence is used for train and validation purposes. We follow the dataset split by 4 of 181 train, 60 validation, 61 test documents. Where our work differs from most prior work is that we utilize the complete subset for test purposes. We use the subset that contains both sufficient and complete annotated evidence, with the same codes, which presents 44 test cases. Since we are interested in cases with more than one evidence span, a filtering step is performed, yielding 17 final test cases. Due to the small amount of data, we additionally use the whole test set, i.e., with sufficient evidence, to analyze model agreement as a rough estimate how reliable the results are. 3.2 Models We rely on training approaches and existing model weights from 6 . The underlying architecture is a transformer, encoder-based architecture, trained on medical text and fine-tuned on MDACE. We use models with two different training regimes. In an unsupervised approach, no evidence annotations are used in the training, but input gradient regularization (IGR) with the aim to reduce the importance of irrelevant tokens. In a supervised approach, human-annotated evidence spans were used in the training process to steer model attention to that evidence. To avoid confusion with the more frequent meaning of supervised in a classification task, we use Evidence-Guided Training (EGT) to refer to this training approach. For each approach 10 seeds from random initializations are available, leading to a total of 20 models. Single models The best model per approach, for IGR and EGT, is determined by computing per-sample winners. Further, we report the maximum value score (max value) as

Read the complete text