Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis

Zhiyin Tan, Jennifer D’Souza · 2026

Read Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis by Zhiyin Tan, Jennifer D’Souza (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2602-10881.txt.

Introduction

Zhiyin Tan and Jennifer D'Souza diagnose where language models fail at evidence extraction for meta-analysis in a 2026 paper. Systematic reviews convert narrative articles into ordered, numerically grounded study records, a task hinging on keeping roles, methods, and effect-size attribution across documents. Whether models meet those structural requirements remains unclear.

The authors propose a structural diagnostic system evaluating extraction as schema-constrained queries of increasing relational and number hardness. Using a manually curated corpus across five scientific domains with a unified query suite, they test two leading models under per-document and long-context multi-document inputs. The system pinpoints failure stages beyond atom-level extraction, showing exactly where ordered evidence work breaks down.

Catalog record

CREATORZhiyin Tan, Jennifer D’Souza
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE61 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2602-10881

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Diagnosing Structural Failures in LLM-Based Evidence Extraction for Meta-Analysis Zhiyin Tan, Jennifer D’Souza Abstract Systematic reviews and meta-analyses rely on converting narrative articles into structured, numerically grounded study records. Despite rapid advances in large language models (LLMs), it remains unclear whether they can meet the structural requirements of this process, which hinge on preserving roles, methods, and effect-size attribution across documents rather than on recognizing isolated entities. We propose a structural, diagnostic framework that evaluates LLM-based evidence extraction as a progression of schema-constrained queries with increasing relational and numerical complexity, enabling precise identification of failure points beyond atom-level extraction. Using a manually curated corpus spanning five scientific domains, together with a unified query suite and evaluation protocol, we evaluate two state-of-the-art LLMs under both per-document and long-context, multi-document input regimes. Across domains and models, performance remains moderate for single-property queries but degrades sharply once tasks require stable binding between variables, roles, statistical methods, and effect sizes. Full meta-analytic association tuples are extracted with near-zero reliability, and long-context inputs further exacerbate these failures. Downstream aggregation amplifies even minor upstream errors, rendering corpus-level statistics unreliable. Our analysis shows that these limitations stem not from entity recognition errors, but from systematic structural breakdowns, including role reversals, cross-analysis binding drift, instance compression in dense result sections, and numeric misattribution, indicating that current LLMs lack the structural fidelity, relational binding, and numerical grounding required for automated meta-analysis. The code and data are publicly available at GitHub. 1 1 1 https://github.com/zhiyintan/LLM-Meta-Analysis [orcid=0009-0002-4166-5810, [email protected], ] *1 [orcid=0000-0002-6616-9509, [email protected], ] keywords Evidence extraction ,Large language models ,Meta-analysis ,Schema-grounded evaluation † † copyrightyear: 2026 † † copyright: Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0). † † venue: IRCDL’25: 22nd Conference on Information and Research Science Connecting to Digital and Library Science, February 19-20 2026, Modena, Italy † † address: L3S Research Center, Leibniz University Hannover, Hannover, Germany † † address: TIB Leibniz Information Centre for Science and Technology, Hannover, Germany † † corresp: Corresponding author. 1 Introduction Systematic reviews and meta-analyses are foundational to evidence synthesis across the sciences, providing structured methods for integrating empirical results at scale 26 ; 15 . Despite advances in digital libraries and scholarly infrastructures, systematic review workflows remain dominated by manual data extraction, where study-level elements, such as populations, variables, measurement units, association methods, and effect sizes, must be identified, normalized, and encoded for analysis 64 . This step is highly labor-intensive and remains a major bottleneck in creating reproducible, machine-actionable meta-analytic datasets. Parallel work in semantic publishing and structured scholarly knowledge has demonstrated that scientific studies can be represented through explicit schemas and ontologies. Examples range from publication-centric models such as SPAR 54 , to domain-specific controlled vocabularies in biomedicine (e.g., MeSH 42 , GO 5 , UMLS 8 ), and large-scale scholarly knowledge graphs such as Semantic Scholar’s Literature Graph 3 and the Computer Science KG 18 . The Open Research Knowledge Graph (ORKG) 6 further shows how paper-level claims and methodological details can be captured via human-authored semantic templates. Large language models (LLMs) appear promising for automating parts of this process. They show strong performance on isolated extraction tasks and domain-adapted scientific text processing 59 ; 16 ; 46 ; 21 , and recent work has explored their use in systematic review screening and semi-automated data extraction pipelines 68 ; 24 . However, a growing body of evidence highlights limitations that are directly relevant to the structural demands of meta-analytic workflows. LLMs struggle with compositional generalization, the systematic combination of familiar elements into novel but valid structures 20 ; 29 . They also exhibit failures in entity and attribute binding, where recognized elements become misaligned or incorrectly associated 22 ; 25 . Long-context processing introduces further fragility, with models often ignoring or misusing information located in the middle of extended inputs 45 . Finally, numerically grounded reasoning remains brittle, particularly when values must be correctly tied to specific variables, units, or statistical methods 2 ; 1 ; 50 . Meta-analysis provides a particularly stringent testbed for these weaknesses. Constructing meta-analytic inputs requires extracting and binding multi-entity, numerically annotated, and method-conditioned tuples, such as population, independent variable, dependent variable, association method, sample size, and effect size, across multiple documents. Success therefore depends not only on recognizing individual entities, but on preserving their structural relationships under aggregation. In this work, we take a structural and diagnostic perspective on LLM-based scientific evidence extraction. Recent work has shown that LLMs can be used for semi-automated evidence extraction in systematic reviews, particularly in biomedical settings 40 . Rather than proposing a new extraction model or end-to-end pipeline, we study how current LLMs behave when evidence extraction is framed as a sequence of increasingly structured, schema-constrained tasks. This framing allows us to explicitly control the relational and binding demands imposed on the model, ranging from isolated entity extraction to higher-order association structures characteristic of meta-analytic inputs. We further consider lightweight aggregation operations to probe whether extracted evidence can be coherently integrated at the corpus level. We instantiate this perspective using a generalized semantic study schema and evaluate it across five scientific domains under controlled multi-document settings. By combining full-text inputs with human-curated gold annotations, our evaluation enables a fine-grained analysis of how model behavior evolves as structural complexity increases. This study makes four main contributions : 1. a structural and diagnostic perspective on LLM-based scientific evidence extraction, which frames extraction as a progression of increasingly constrained, schema-guided tasks rather than as an end-to-end pipeline problem; 2. a tiered query framework that operationalizes this perspective through controlled variations in tuple arity and relational binding complexity, covering both direct extraction and derived statistical reasoning; 3. an manually curated, multi-domain, schema-aligned evaluation benchmark spanning five scientific domains (civil engineering, medical and health science, agricultural science, earth and environmental sciences, and social science) designed to systematically investigate LLM performance under increasing structural demands; 4. a comprehensive empirical analysis identifying where along this structured progression modern LLMs begin to fail, revealing specific weaknesses in relational binding, numerical grounding, and document-level attribution. Together, these contributions clarify the structural requirements of automated meta-analytic evidence extraction and provide a foundation for designing future LLM-based systems that better align with the needs of evidence synthesis and digital library infrastructure. 2 Related Work Semantic Modeling and Structured Representations of Scientific Studies. Prior work in semantic publishing treats scientific articles as structured objects rather than purely narrative text, modeling key study components such as populations, variables, interventions, outcomes, and measurements using explicit schemas and ontologies 53 ; 14 ; 13 . This perspective enables machine-interpretable access to scientific content beyond surface-level text. This idea has been instantiated at scale through ontology suites and scholarly knowledge graphs that formalize document structure, citation relations, and scientific statements 53 ; 54 ; 13 ; 35 ; 27 . The Open Research Knowledge Graph further demonstrates paper-level contribution modeling via human-authored schemas 49 , while autonomously constructed graphs such as MatKG show that complex scientific entities and relations can be populated at corpus scale 65 . Recent work also explores LLM-assisted schema discovery and refinement 57 . In this setting, a central question is how such structured study representations

Read the complete text