An Auditable Pipeline for Fuzzy Full-Text Screening in Systematic Reviews: Integrating Contrastive Semantic Highlighting and LLM Judgment
P. Mortezaagha, A. Rahgozar · 2025
Read An Auditable Pipeline for Fuzzy Full-Text Screening in Systematic Reviews: Integrating Contrastive Semantic Highlighting and LLM Judgment by P. Mortezaagha, A. Rahgozar (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-15822.txt.
Introduction
Pouria Mortezaagha and Arya Rahgozar built a screening pipeline for the slow step of systematic reviews: reading full texts to judge if a study fits. Key proof hides in long, mixed papers, so hard yes-no rules fail. Their pipeline treats the call as a graded one. It also keeps a trail so each call can be checked later.
Papers are cut into chunks and matched against each rule for the review. A fuzzy logic step turns the match scores into graded in-or-out degrees. A large language model then judges the marked spans and gives grounds for each call. On a pilot set of sixteen full texts, the pipeline caught far more true fits than two base methods. It also cut the time per paper from about twenty minutes to under one.
Catalog record
| CREATOR | P. Mortezaagha, A. Rahgozar |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-15822
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
An Auditable Pipeline for Fuzzy Full-Text Screening in Systematic Reviews: Integrating Contrastive Semantic Highlighting and LLM Judgment P. Mortezaagha, A. Rahgozar Abstract Full-text screening is the major bottleneck of systematic reviews (SRs), as decisive evidence is dispersed across long, heterogeneous documents and rarely admits static, binary rules. We present a scalable, auditable pipeline that reframes inclusion/exclusion as a fuzzy decision problem and benchmark it against statistical and crisp baselines in the context of the Population Health Modelling Consensus Reporting Network for noncommunicable diseases (POPCORN). Articles are parsed into overlapping chunks and embedded with a domain-adapted model; for each criterion (Population, Intervention, Outcome, Study Approach), we compute contrastive similarity (inclusion–exclusion cosine) and a vagueness margin , which a Mamdani fuzzy controller maps into graded inclusion degrees with dynamic thresholds in a multi-label classification setting. A large language model (LLM) judge adjudicates highlighted spans with tertiary labels, confidence scores, and criterion-referenced rationales; when evidence is insufficient, fuzzy membership is attenuated rather than hard-excluded. In a pilot on an all-positive gold set ( N = 16 N{=}16 full texts; M = 3,208 M{=}3{,}208 chunks), the fuzzy system achieved document-level recall of 81.25% (Population), 87.50% (Intervention), 87.50% (Outcome), and 75.00% (Study Approach) with 95% Wilson confidence intervals, exceeding statistical baselines (recall 56.25–75.00%) and crisp baselines (recall 43.75–81.25%). Strict “all-criteria” inclusion was reached for 50.00% of articles, compared to 25.00% and 12.50% under the baselines. Cross-model agreement on justifications was 98.27%, human–machine agreement 96.07%, and a pilot review showed 91% inter-rater agreement ( κ = 0.82 \kappa{=}0.82 ) with screening time reduced from ∼ \sim 20 minutes to under 1 minute per article at a significantly lower cost. These results demonstrate that fuzzy logic, combined with contrastive highlighting and explainable LLM adjudication, delivers high recall, stable rationale, and end-to-end traceability. Because this proof-of-concept used only positives, the evaluation emphasized recall. \keepXColumns Keywords: Systematic review automation, Full-text screening, Fuzzy logic (Mamdani), Multi-label text classification, Explainable AI, Large Language Models (LLMs), Contrastive semantic similarity, Biomedical NLP, Evidence synthesis 1 Introduction Systematic reviews (SRs) are the cornerstone of evidence-based medicine and biomedical research, providing the most reliable synthesis of scientific knowledge for clinical, policy, and research decisions [ Tawfik2019sr ] . Their rigor rests on pre-specified questions, comprehensive searches, explicit inclusion/exclusion criteria, and transparent appraisal. Yet the scale of contemporary publishing has overwhelmed traditional workflows: recent estimates indicate more than 75 clinical trials and 11 SRs appear each day [ 3 ] , and the pool of potentially eligible studies has expanded far beyond available reviewer capacity [ 2 , 4 ] . In practice, teams now screen thousands of citations and frequently advance hundreds to full text [ 5 , 6 ] . Full-text screening as the bottleneck. Unlike title/abstract triage, full-text assessment demands careful reading of long, heterogeneous documents. Decisive information may be scattered across sections, buried in tables/figures, or described in discipline-specific or evolving terminology [ 8 , 7 ] . The task is labor-intensive and error-prone; reviewer fatigue and cognitive overload are common, and—even with shared protocols—experts often disagree on eligibility at this stage [ 7 ] . This disagreement reflects not only human inconsistency but also a property of the data: full-text evidence is inherently ambiguous, context-dependent, and, in a precise sense, fuzzy . 1.1 The Inherently Fuzzy Nature of Full-Text Decision-Making By “fuzzy” we mean that signals supporting inclusion/exclusion are graded, partial, and distributed. Populations are described broadly in one section and narrowly in another; interventions may be referenced indirectly; outcomes are reported with non-standard or shifting terminology. Criteria are seldom satisfied in an all-or-nothing manner; rather, eligibility emerges from partial matches and multi-dimensional semantic relationships. Consequently, two diligent reviewers can reasonably reach different conclusions on the same article, with consensus achieved only after discussion that weighs context and degree—a phenomenon well documented for full-text screening [ 7 ] . 1.2 Limits of Traditional and Probabilistic Approaches Crisp rule-based systems are explicit and transparent but poorly matched to graded, contextual evidence; they cannot represent partial relevance or uncertainty [ 11 , 7 ] . Probabilistic machine learning offers improvements but still reduces decisions to thresholds on opaque scores and often lacks persuasive, case-specific explanations [ 5 , 6 , 8 ] . Recent advances in large language models (LLMs) improve coverage and flexibility, yet LLMs alone do not guarantee consistent recall at SR standards, nor do they natively provide calibrated, graded decisions for the “gray-zone” cases that dominate full-text assessment [ 13 , 14 ] . 1.3 Why Fuzzy Logic is Needed Fuzzy logic [ 18 ] offers a human-aligned principled calculus to represent degrees of membership and partial truth [ 9 , 10 , 11 , 12 ] . In the context of full-text screening, much of the challenge arises not from simple ambiguity, which implies multiple possible interpretations, but from vagueness , where key eligibility signals are imprecise, graded, or previously nonmeasurable [ 32 ] . Fuzzy logic rectifies this vagueness by transforming qualitative, diffuse evidence into quantifiable degrees of support, enabling systematic comparison across heterogeneous text spans [ 32 ] . It encodes the interaction of signal strength and contextual nuance in a transparent rule base [ 28 ] , mirroring how expert panels reason: weighing multiple, partially supportive statements until a consensus degree of inclusion is reached. Crucially, fuzzy systems are both interpretable and auditable, allowing users to inspect membership functions, rule activations, and the exact path from text to decision [ 19 , 20 ] . 1.4 Innovations in Fuzzy, Explainable, and LLM-Augmented Screening We propose a full-text screening pipeline designed from first principles for fuzziness and explainability: • Contrastive semantic highlighting. Articles are segmented and embedded; for each criterion, we compute inclusion-versus-exclusion similarity and highlight candidate spans, making the evidence traceable to concrete text. • Mamdani fuzzy inference. A Mamdani controller aggregates similarity strength and an explicit vagueness margin to produce graded inclusion degrees and support multi-label assignment [ 12 ] . • LLM-as-judge. A domain-adapted LLM interprets the fuzzy-selected spans, issues a tertiary judgment with a confidence score, and provides a short, criterion-referenced explanation to enhance transparency and arbitrate borderline conflicts [ 17 , 13 , 14 , 21 , 22 ] . • Handling lack of evidence. When highlighted spans are judged insufficient at a later stage, their contribution is attenuated rather than forcing a brittle exclude/retain decision, preserving recall while demoting weak evidence. This architecture targets high recall with bounded reviewer effort and an auditable chain from document text to decision. 1.5 Related Work and Comparative Approaches Study selection methods span crisp manual tools, probabilistic ML, fuzzy logic, and LLM-based prompting. Table 1 contrasts representative systems, highlighting strengths and limitations across precision/recall, transparency, and scalability [ 7 , 5 , 8 , 6 , 12 , 13 , 14 ] . Our contribution sits at the intersection: a fuzzy, explainable controller layered on modern NLP with LLM adjudication. Table 1: Comparison of approaches for systematic review screening. Approach Examples Strengths Limitations Crisp manual / rule-based Covidence, Rayyan, keyword filters [ 7 ] High precision, explicit criteria, transparent; widely used in practice Labor-intensive, inconsistent on ambiguous cases, no partial relevance, “maybe” still needs resolution Probabilistic ML Abstrackr (SVM), BERT-based classifiers [ 5 , 8