Evaluating Elicit’s systematic reviews workflow in an umbrella review on air pollution and acute lower respiratory infections: a methodological study for quality appraisal Authors: Cristina Mazzali, Tiziana Pinciroli, Maria Rosa Valetto, Pietro Dri, Antonio Giampiero Russo, Paola Angelini, Paola Angelini, Serena Broccoli, Marco Monti, Nicola Caranci, Katia Raffaelli, Daniela Fortuna, Paolo Giorgi Rossi, Marta Ottone, Laura Bonvicini, Luisa Frova, Stefano Marchetti, Lidia Gargiuolo, Andrea Ranzi, Simone Giannini, Stefano Marchesi, Annamaria Colacci, Francesca Russo, Vanessa Groppi, Luisa Memo, Sofia Memoli, Gisella Pitter, Elena Narne, Giulia Capodaglio, Manuel Zorzi, Nicola Gennaro, Laura Cestari, Rodolfo Bassan, Massimo Bressan, Luca Zagolin, Nicoletta Cornaggia, Veronica Todeschini, Matteo Lazzarini, Rossella Murtas, Michele Carugno Published: 2026 ABSTRACT BackgroundThe exponential growth of scientific publications has increased the complexity of evidence synthesis. Systematic reviews remain essential but highly resource-intensive. Large language models (LLMs) offer new opportunities to support or partially automate key steps of this process. This study evaluates the performance of Elicit’s Systematic Reviews workflow in comparison to the traditional methodology, using as reference a published umbrella review on the association between air pollution and acute lower respiratory infections (ALRI).MethodsA parallel workflow was developed to reproduce each phase of the traditional review. Considering article retrieval, for the traditional workflow, articles were retrieved through a Boolean search, while for the AI-assisted workflow a natural-language query submitted to Elicit. Screening was conducted in two steps, emulating the traditional PECOS-based criteria. Full-text evaluation and quality appraisal were performed through Elicit’s “data extraction” functionalities. For quality appraisal the validated AMSTAR-2 EH questionnaire was applied.ResultsThe traditional Boolean search identified 324 unique articles. When compared with the 500 records retrieved by Elicit, an overlap of 8% was observed, which prevented a direct, recall-oriented comparison of search performance. To enable a controlled comparison of downstream steps, we applied Elicit’s screening and data-extraction functions to the 324 records identified through the Boolean search, using empirically defined screening-score threshold in Elicit to select studies for further evaluation. In the screening on title and abstract, 33 articles were identified through the traditional workflow and 70 through Elicit, 30 articles overlapping (recall 90.9%, precision 42.9%). The full-text screening selected 15 articles with the traditional methodology and 24 with Elicit, all 15 articles from the traditional methodology being included in Elicit selection (recall 100%, precision 62.5%).In the quality assessment, Elicit showed 24.4% disagreement on general items and 30.4% on additional items of the AMSTAR-2 EH. Errors clustered around multi-component questions, items requiring expert interpretation, and information located in supplementary materials.ConclusionsElicit can support several phases of systematic reviews and reduce manual workload, but it cannot independently reproduce the methodological rigor required for high-quality evidence synthesis. At present, LLM-based tools are best positioned as complementary systems within human-supervised workflows.Supplementary InformationThe online version contains supplementary material available at 10.1186/s12874-026-02892-3. Background The exponential growth of medical and scientific literature has made evidence synthesis an essential tool for summarizing and integrating findings across studies. Systematic reviews play a pivotal role in providing up-to-date and transparent summaries of evidence, enabling reproducibility and accessibility in clinical decision-making [1–3]. However, these reviews are notoriously time-consuming and resource-intensive. Conducting a systematic review demands substantial time, resources, and financial investment [4–8]. Among the wide range of applications of artificial intelligence (AI) in medicine, large language models (LLMs) opened new perspectives for improving the efficiency of evidence synthesis [9–11]. LLMs, trained to perform complex language modeling tasks, can assist in automating various stages of the review process, from screening to data extraction. Their application has already shown substantial time savings, for instance, reducing screening workload by up to 77% in some studies [1]. Although their use still requires rigorous validation, LLMs represent a promising step toward accelerating systematic reviews and making evidence synthesis more scalable in an era of information overload [12, 13]. This work is framed within the Project “Aria outdoor e salute: un atlante integrato a supporto delle decisioni e della ricerca”/“Outdoor Air and Health: an Integrated Atlas to Support Decision-Making and Research”, which is funded by the Ministry of Health within the “Piano Nazionale per gli investimenti complementari” (PNC). Within the PNC Project the Epidemiology Unit of the Agency for Health Protection (ATS) of Milan aims to synthesize the available evidence about health endpoints for which the relationship with air pollution has not yet been well established. Examples of such endpoints can be found in the World Health Organization (WHO) latest Air Quality Guidelines report (2022) [14]. The goal of this methodological study was to compare an AI assisted workflow with a traditional umbrella review process, using a published umbrella review on air pollution and Acute Lower Respiratory Infections (ALRI) as a reference framework, [15] with the ultimate aim of refining a tool that can later be applied to health endpoints for which the relationship with air pollution is not yet well established. The review used a reference based on a predefined PECOS and followed an internal protocol based on PRISMA methodology, shared and agreed upon with an expert panel. The first screening phase was conducted by 4 researchers, and the second phase by 13 researchers. In both phases, reviewers worked independently in pairs. The quality of selected papers was critically appraised using the AMSTAR2-EH tool [16]. In particular, the Systematic Reviews workflow of Elicit, a tool based on LLMs capable of selecting and organizing information from scientific literature, was tested in the steps of article search and screening according to a predefined search strategy, as well as in the quality evaluation of the selected articles following the criteria of a standardized and validated questionnaire such as AMSTAR-2 EH [16]. Methods Artificial intelligence tool elicit Elicit is a research tool developed and incubated by Ought (USA) since 2018, released in 2019, and transformed in 2023 into an independent Public Benefit Corporation. It is based on LLMs capable of selecting and organizing information from scientific literature [17]. Elicit searches through more than 138 million academic articles within the Semantic Scholar corpus, which includes publications from databases such as PubMed, IEEE Xplore, ACM Digital Library, Springer Nature, Wiley Online Library, and others, but does not include Embase. The Systematic Reviews workflow of Elicit was released in 2025 and provides functionalities able to support almost all major steps of the systematic review process. In this paper, the term Elicit specifically refers to the Systematic Review workflow, unless otherwise specified. As Elicit does not support Boolean operators, article selection is carried out through a natural language query that searches for meta-analyses, systematic reviews, or meta-regressions. At the moment our research was conducted the tool (in its “Pro” subscription version) allowed users to select up to 500 papers per search, retrieved via its internal search engine and/or uploaded by the user. Moreover, further 500 papers could be uploaded, for a total of 1,000 papers per review. Data are organized into predefined or user-defined concept groups to support screening, extraction, and synthesis within the platform. Elicit provides short rationales for screening and extraction outputs and, when available, highlights supporting reasoning and excerpts from the record; these outputs can be exported as CSV files (Additional file 1). This characteristic, more specifically the ability to present this reasoning in textual form (few lines organized in columns, accompanied by direct citations from the article’s text), was considered valuable and ultimately led to the decision to use Elicit among other candidate tools. All Elicit runs were performed between September 2025 and October 2025. To improve reproducibility, we report the access date, subscription tier, workflow used, and the exact run settings (including filters) in the additional materials (Additional file 2). Workflow We developed a workflow parallel to that used when adopting the traditional methodology encompassing the extraction, selection, analysis, evaluation, and compilation activities aimed at assessing the association between exposure to air pollutants and acute lower respiratory infections (ALRI). In addition, to validate the process, we performed a step-by-step comparison of the AI-driven and traditional workflows (Fig. 1).Fig. 1Step-by-step comparison of the AI-driven and traditional workflows Step-by-step comparison of the AI-driven and traditional workflows Step 1—Definition of the natural language query Several tests were conducted to define the query for article retrieval in natural language analogous to the Boolean one [15]. This process consisted of fine-tuning the natural language query to prompt Elicit, taking into account the keywords and Boolean operators of the original PECOS question used for the systematic review [15]. For this purpose, two authors (CM and MRV) worked independently on the natural language query and resolved any discrepancies by consensus. The final query submitted to Elicit was: “Find meta-analysis or systematic reviews or meta regressions about exposure to air pollution or air pollutants, or air quality or air particulates or environmental pollutants or environmental pollution or urban pollution or particulate matter or PM2.5 or PM10 or PM2.5–10 or coarse particulate or particulate air or suspended particles or coarse particles or carbon monoxide or nitrogen dioxide (NO2) or nitrogen oxides or sulfur dioxide (SO2) or sulfur oxides or ozone (O3) and acute lower respiratory infection (ALRI) or acute lower respiratory tract infection or lower respiratory tract infection (LTRI) or Pneumonia or Bronchopneumonia or Pleuropneumonia or bacterial pneumonia or necrotizing pneumonia or pneumocystis pneumonia or viral pneumonia or Bronchiolitis or Bronchitis.” Step 2 – Article search A dual-path approach was adopted:article search among the articles directly retrieved by Elicit (N ≤ 500), applying the filter study type = systematic review, but not the filter study type = meta-analysis, as the tests conducted showed that it tended to include primary studies, while its exclusion did not eliminate any of the studies of interest. This conclusion is based on the empirical comparison of the two sets of retrieved records obtained with and without the term “meta-analysis”. The “Publication date” filter was only partially implemented at the start of the study; therefore, we could specify only a starting year and an ending year for eligibility, consistently with the time window used in the reference umbrella review. When we performed the article search Elicit ranked the up to 500 articles by plausibility based on natural language interpretation.We also imported the records retrieved through the traditional Boolean search by uploading two plain-text files provided by ATS Milan, which contained bibliographic metadata for records identified in PubMed and Embase. article search among the articles directly retrieved by Elicit (N ≤ 500), applying the filter study type = systematic review, but not the filter study type = meta-analysis, as the tests conducted showed that it tended to include primary studies, while its exclusion did not eliminate any of the studies of interest. This conclusion is based on the empirical comparison of the two sets of retrieved records obtained with and without the term “meta-analysis”. The “Publication date” filter was only partially implemented at the start of the study; therefore, we could specify only a starting year and an ending year for eligibility, consistently with the time window used in the reference umbrella review. When we performed the article search Elicit ranked the up to 500 articles by plausibility based on natural language interpretation. article search among the articles directly retrieved by Elicit (N ≤ 500), applying the filter study type = systematic review, but not the filter study type = meta-analysis, as the tests conducted showed that it tended to include primary studies, while its exclusion did not eliminate any of the studies of interest. This conclusion is based on the empirical comparison of the two sets of retrieved records obtained with and without the term “meta-analysis”. The “Publication date” filter was only partially implemented at the start of the study; therefore, we could specify only a starting year and an ending year for eligibility, consistently with the time window used in the reference umbrella review. When we performed the article search Elicit ranked the up to 500 articles by plausibility based on natural language interpretation. We also imported the records retrieved through the traditional Boolean search by uploading two plain-text files provided by ATS Milan, which contained bibliographic metadata for records identified in PubMed and Embase. We also imported the records retrieved through the traditional Boolean search by uploading two plain-text files provided by ATS Milan, which contained bibliographic metadata for records identified in PubMed and Embase. Step 3 – Article screening The article screening process was planned in two steps in accordance with the traditional methodology, first on title and abstract and subsequently on full-text (Figs 2 and 3).Fig. 2Article screening on title and abstractFig. 3Article screening on full-text Article screening on title and abstract Article screening on full-text Considering the screening on title and abstract, it was conducted using the “screening” functionality of Elicit on the base of the inclusion and exclusion criteria of the traditional methodology (PECOS question) reformulated in natural language. Since Elicit does not support exclusion criteria, these were reformulated as inclusion criteria as appropriate. The screening criteria used in the screening step on title and abstract are available in the Additional file 3. To select articles that pass the screening a threshold was then set in the system. Elicit assigns a score based on the consistency between each article and the PECOS criteria, with higher scores (up to a maximum of 5) indicating a stronger consistency. By default, the tool proposes a threshold positioned in the middle of the scoring range, i.e., 2.5 which can be manually adjusted: the higher it is set, the fewer articles will be selected during screening. To define the threshold applied in this study, two combined empirical criteria were adopted: analysis of the score distribution provided by Elicit and inclusion of approximately 20% of the initial pool of articles. It was hypothesized that the plausibility-based selection criterion used by Elicit could identify a characteristic distribution (e.g., clustering or a clear demarcation between relevant and non-relevant articles). The criterion requiring the inclusion of at least 20% of the articles is to be regarded as precautionary in nature. This approach represents a generalizable methodology for article selection, applicable without prior knowledge of the results from traditional search strategies, and which, under different conditions, may yield different numerical thresholds. For the screening on full-text of the articles selected after the screening on title and abstract, the “data extraction” functionality of Elicit was used, which manages both inclusion and exclusion criteria. When the full-text was not available in Elicit, it was uploaded manually. The choice of the “data extraction” functionality was mandatory, as the “screening” functionality does not operate on full-texts, and its use in the second screening step would have deviated from the traditional workflow. To perform the screening on full-text, a format derived from the predefined ATS screening criteria was implemented (Additional file 4). In particular, for each element of the PECOS framework, one inclusion criterion and one exclusion criterion were created. The criteria were the same as those adopted for the screening on title and abstract, except that the exclusion criteria were reformulated in a more direct manner and a binary Yes/No response was forced. Step 4 – Application of the AMSTAR-2 EH criteria to analyze the selected articles The quality of the article selected was assessed on the basis of the AMSTAR-2 EH questionnaire, an adaptation of the AMSTAR-2 tool [18] introduced and validated by Forastiere et al. [16] to evaluate the methodological quality of systematic reviews in environmental epidemiology. The tool is composed of 12 items derived from AMSTAR-2 and 9 new items specifically conceived for environmental epidemiology. The authors typically assess the first 12 items, and only evaluate the remaining 9 if predefined quality criteria are met [15, 16, 18]. AMSTAR-2 EH questions were reformulated in natural language (Additional file 5). As Elicit fills up to 20 items and AMSTAR-2 EH includes 21 items; the last one, easily identifiable by the operator, was not processed by Elicit. For each assessment, Elicit provided the dichotomic answer (Yes/No) to the AMSTAR-2 EH, reasoning, supporting quote, and/or supporting table exportable as a.csv file. The answers provided by Elicit were compared to those of the traditional methodology (Independent evaluation by two reviewers, with any divergent judgment resolved by the evaluation of a third reviewer). Comparison between Elicit-driven and traditional workflows We compared Elicit-driven and traditional workflows at three checkpoints. After Step 2, we quantified the overlap between retrieval sets. After Steps 3.1 and 3.2, we compared screening decisions by treating the human reviewers’ final decisions as the reference standard and reporting the number of records retained by each approach as well as agreement, recall, and precision, which confidence intervals (CI) were calculated based on the binomial distribution. After Step 4, we compared AMSTAR-2 EH answers item-by-item and summarized disagreement rates overall and for the items most frequently misclassified by Elicit. Results Step 1—Definition of the natural language query Elicit automatically evaluates the clarity and effectiveness of the query. In this case, the query received a “Good research question” rating. However, as Elicit does not explicitly disclose the criteria adopted, we did not take into account that evaluation. Elicit also suggested additional specifications, which were intentionally not implemented to maintain consistency with the original Boolean query. Step 2—Article search Elicit returned a list of 500 articles matching the query (available upon request). The Boolean search retrieved 325 records (158 from PubMed and 167 from Embase). After excluding one unusable record, 324 records were retained for screening. The overlap between the two sets (the 500 articles retrieved by Elicit and the 324 articles identified through the Boolean query) was only 8% (n = 40) which further decreased to 6.8% (n = 34) when considering only the PubMed articles (n = 158). Because overlap alone cannot be used to estimate recall without an external gold standard, and because the observed overlap was low, we conducted Step 3 exclusively on the 324 records retrieved through the traditional approach to ensure a controlled, same-input comparison of screening and appraisal performance. Step 3—Article screening Screening on title and abstract Among the 324 records retrieved through the traditional methodology, Elicit assigned a minimum screening score of 0.1 to 124 records (38.3%). Scores of 1.4 and 2.5 were assigned to 90 (27.8%) and 40 records (12.3%), respectively. An additional 25 records (7.7%) received a score of 2.7. The distribution of the articles helped inform the threshold setting as a clustering pattern emerged with a somewhat distinct separation and three numerous clusters with lower scores (excluded). We set the screening-score threshold at 2.6, retaining 70 records for full-text assessment. At this stage, Elicit retained 30 of the 33 records selected by human reviewers, corresponding to an agreement of 86.7% (95% CI 82.5–90.2), a recall of 90.9% (95% CI 75.7–98.1) and a precision of 42.9% (95% CI 31.1–55.3) when using the human selection as reference. (Additional files 6 and 7). The 3 records, which were not retained, received scores of 2.5 (one article) and 1.4 (the other two). It should be noted, however, that none of these three articles subsequently passed the full-text selection stage in the traditional workflow. Screening on full-text Of the 70 full-text articles selected by Elicit, one could not be uploaded due to a platform error. Therefore, only 69 articles were subjected to the second sub-step of screening, which involved extracting information related to the inclusion and exclusion criteria (Additional file 8). This second screening resulted in the final selection of 24 articles to be submitted to AMSTAR-2 EH. Notably, all 15 articles that had been identified through the traditional methodology for AMSTAR-2 EH evaluation were included in the final selection of 24 articles (recall 100%, 95% CI 78.2–100.0; precision 62.5%, 95% CI 40.6–81.2). Of the 9 additional studies identified by Elicit, 4 had been excluded in the title and abstract screening step and 5 in the full-text screening step of the traditional methodology. Upon individual analysis, their inclusion was deemed incorrect for the following reasons, with more than one reason applying to 5 of the 9 studies: incorrect exposure (n = 6), incorrect comparator (n = 4), incorrect outcome (n = 3), and incorrect study type (n = 3). Step 4—Application of AMSTAR-2 EH criteria Elicit’s responses were largely consistent with reviewers’ ratings, though not identical (Table 1).Table 1Elicit responses to AMSTAR-2 EH questionnaireGeneral questionsAdditional questionsAnswers123456789101112TotalA1A2A3A4A5A6A7A8TotalTotal1515151515151515151515151807777777756NA00000000000008888888864Correct151311131391013131013131466757224639Incorrect024226522522341020553117Resolved by consensus among correct01301310211114001000214Resolved by consensus among incorrect00200130120110000000000NA indicates items that were not assessed by human reviewers following the traditional methodology Table 1Elicit responses to AMSTAR-2 EH questionnaireGeneral questionsAdditional questionsAnswers123456789101112TotalA1A2A3A4A5A6A7A8TotalTotal1515151515151515151515151807777777756NA00000000000008888888864Correct151311131391013131013131466757224639Incorrect024226522522341020553117Resolved by consensus among correct01301310211114001000214Resolved by consensus among incorrect00200130120110000000000NA indicates items that were not assessed by human reviewers following the traditional methodology Elicit responses to AMSTAR-2 EH questionnaire NA indicates items that were not assessed by human reviewers following the traditional methodology Among the 180 answers provided by Elicit for the 12 general AMSTAR-2 EH questions across 15 papers, 44 were incorrect (24.4%). For 7 out of 15 articles, the reviewers evaluated all the 8 additional AMSTAR-2 EH items, resulting in 56 answers. Seventeen of the Elicit answers were incorrect (30.4%). The errors were concentrated in a few questions, namely the general AMSTAR-2 EH questions 6, 7, and 10, and the additional AMSTAR-2 EH questions 5 and 6. Some disagreements between Elicit’s answers and those provided by the reviewers corresponded to questions that the reviewers had resolved by consensus. Specifically, Elicit gave 10 incorrect answers to general questions that the reviewers had resolved by consensus. However, Elicit also gave 14 correct answers to general questions and 4 correct answers to additional questions that the reviewers had resolved by consensus (Table 1 and Additional file 9). A situation that can hinder Elicit’s ability to provide accurate answers occurs when the relevant information is contained in the supplementary materials. Elicit mentioned supplementary materials when addressing 27 AMSTAR-2 EH questions related to 8 full-text papers. We registered one hallucination related to Chen et al. [19]. For AMSTAR-2 EH item 12, Elicit incorrectly answered ‘No’ and justified the answer by stating that competing interests or funding sources were not reported, despite this information being explicitly present in the full text. Discussion Systematic reviews can support the task of managing evidence, which is becoming increasingly complex as medical and scientific research output has grown exponentially [1, 20]. They can help synthesize this expanding body of evidence into transparent and reproducible summaries of existing research [1–3, 21, 22], and can further improve in efficiency, optimization, and rigor through the integration of AI tools with human expertise [23]. On the other hand, systematic reviews constitute one of the most resource-intensive activities in evidence-based medicine, requiring substantial investment in time, expertise, and financial resources due to the necessity of screening vast quantities of literature while maintaining strict methodological standards and completeness [1, 7, 8, 12]. It has been reported that conducting a single systematic review may require more than 1,000 h of highly skilled work [4], take on average 67 weeks from registration to publication [5], or to span overall completion times of approximately six months to two years [24]. When considering the cumulative effort of information specialists, methodologists, and subject-matter experts across all phases, total costs may surpass 100,000 USD [6]. Within this workflow, data extraction represents one of the most critical and demanding stages. It requires between 107 and 172 min per study, depending on whether extraction is performed by a single reviewer with verification or through dual independent extraction [25, 26]. In light of these constraints, integrating AI-based solutions can help meet more innovative and efficient strategies to address the escalating demands of systematic review production [23, 27]. Evidence indicates that AI-assisted screening can substantially reduce the manual burden, with reports showing that, following AI implementation, only 23% of retrieved records required human screening, translating into a 77% reduction in title and abstract screening time [1]. In particular, AI applications based on LLMs seem to offer new perspectives for improving the efficiency and accuracy of evidence synthesis within systematic reviews as tools to support or partially automate key phases of the process [8, 10–12, 22]. A relevant advantage lies in the possibility to avoid specific training when applied to new reviews, although fine-tuning remains possible [12, 22]. LLMs can assist in developing and evaluating search strategies, automating screening of titles and abstracts, extracting structured data, supporting risk-of-bias assessments, translating non-English studies, and drafting standardized sections of reports, with their behavior being adaptable through prompt engineering and parameter tuning [12]. Considering screening tasks, non-customized models such as GPT-4, DeepSeek and Phi-4, achieved a 40% reduction in workload while identifying more than 90% of eligible studies (recall above 0.9). [28]. Similarly, ChatGPT-3.5 has shown high sensitivity (95%) and Negative Predictive Value (NPV, 99%), suggesting effectiveness in excluding clearly irrelevant studies while supporting a hybrid workflow in which humans assess borderline cases [29]. Regarding screening, one study evaluated the performance of AI tools (ChatGPT 4.0, Claude 3.5, Gemini 1.5, DeepSeek-V3, and RobotSearch) on a pool of articles preselected by experts and found that these tools have limited diagnostic accuracy in distinguishing between randomised clinical trials (RCTs) and non-RCTs. The False Negative Fraction (FNF) for RCTs ranged from 6.4% to 13.0%, while the False Positive Fraction (FPF) for non-RCTs ranged from 2.8% to 22.2% [30]. For data extraction, LLMs have shown accuracy rates between 80 and 96%, although such findings are considered preliminary [10, 13]. In risk-of-bias assessment, results remain inconsistent, with agreement levels ranging from slight to substantial when compared to human judgments, indicating that LLMs may be more suitable for supporting or double-checking this task rather than replacing expert reviewers [12, 31, 32]. On the other hand, LLM data sources are often opaque, conflicting with the principles of transparency and traceability fundamental to evidence-based research [12, 33]. Their outputs are frequently difficult to reproduce, particularly due to continuous model updates and the variability of generative responses [12, 28]. Furthermore, the phenomenon of “hallucinations,” poses serious concerns for their unsupervised use [8]. A scoping review mapped how LLMs are being used in health research to support systematic reviews. The analysis covered 37 studies published between 2021 and 2024, showing that LLMs, mostly GPT-based, have been tested in most review stages. While about half of the studies reported encouraging results, many remained inconclusive or negative suggesting that LLMs are promising, but at the moment their use needs human supervision due to ongoing concerns about accuracy, reproducibility, and methodological robustness [34]. A similar conclusion was reached in a systematic review of 19 studies comparing traditional expert-driven approaches to evidence synthesis with generative AI tools (ChatGPT, GPT, Claude, Bing AI, and Perplexity.AI). The review found a high or unclear risk of bias in key areas such as review selection, generative AI conduct, and applicability of results. Specifically, these tools missed 68% to 96% of relevant studies during the search phase (median 91%) and made incorrect inclusion/exclusion decisions (incorrect inclusion: range 0–29%, median 10%; incorrect exclusion: range 1–83%, median 28%). Additionally, there were errors in data extraction (range 4–31%, median 14%) and risk-of-bias assessments (range 10–56%, median 27%). These findings suggest that while GenAI shows potential in assisting with certain evidence synthesis tasks, human oversight is essential, particularly in the areas of search and screening [35]. Among LLM-based tools, Elicit [17] is specifically designed to support literature searches. Beyond the absence of training requirements, which is common to all LLMs, its key feature is the orientation toward bibliographic research, with dedicated functionalities for evidence retrieval and screening. In 2025, the Systematic Reviews workflow in Elicit has been empowered, and several updates and improvements have been released resulting in a continuous refinement of the tool, driven by ongoing interactions and feedback between developers and users, but also potentially limiting reproducibility over time. When considering the selection of articles, it has to be pointed out that Elicit adopts a search methodology that differs fundamentally to the traditional as it does not rely on Boolean logic but on ranking up articles according to their plausibility based on natural language interpretation and extends its queries to sources beyond PubMed, though not to Embase. As a result, the direct comparison between Elicit-driven search and traditional searches is subject to certain limitations. The overlap < 10% observed between the two approaches was deemed insufficient on its own to determine that Elicit captured all studies identified through the traditional search. This very low overlap cannot, at present, be explained with certainty due to both the limited transparency of the underlying mechanisms of Elicit and the still limited experience with its application in the context of systematic reviews. In this regard, only two previous studies have evaluated the sensitivity of Elicit in retrieving studies identified through traditional search strategies, reporting values of 17.6% in a single-case evaluation [23] and 39.5% (range: 25.5–69.2%) across multiple case studies [36]. Although these values are not optimal, they are higher than those observed in the present study. An additional hypothesis concerns the nature of the end-points explored. Non-consolidated end-points (i.e., those for which the relationship with air pollution is not yet well established) may pose greater challenges for LLM-based tools such as Elicit than more established end-points. As these models rely on semantic similarity [23], less-established relationships — often described using heterogeneous or less frequent terminology — may be less reliably captured, potentially introducing semantic biases. This issue is relevant for the future development of systematic review methodologies supported by LLMs and deserves further investigation. At present, the extent to which a tool such as Elicit can fulfill the function of article search remains uncertain and subject to debate and further verification. An additional hypothesis concerns the nature of the end-points explored. Non-consolidated end-points (i.e., those for which the relationship with air pollution is not yet well established) may pose greater challenges for LLM-based tools such as Elicit than more established end-points. As these models rely on semantic similarity [23], less-established relationships—often described using heterogeneous or less frequent terminology—may be less reliably captured, potentially introducing semantic biases. This issue is relevant for the future development of systematic review methodologies supported by LLMs and deserves further investigation. At present, the extent to which a tool such as Elicit can fulfill the function of article search remains uncertain and subject to debate and further verification. One of Elicit’s strengths lies in its use of graded inclusion criteria rather than binary selection, providing ranked relevance scores. On the other hand, the lack of transparency regarding how these scores are computed currently limits explainability. Elicit lacks repeatability per se – a limitation inherent to LLM-based tools, whose outputs vary across runs due to probabilistic nature in token generation which can lead to different results on repeated attempts [37]. It means that identical inputs may yield different outputs across runs. However, even the traditional human-led systematic reviews are not perfectly repeatable, as independent reviews on the same research question may yield different sets of included studies and divergent conclusions [23]. A further limitation of Elicit is that it draws exclusively from the Semantic Scholar database, which restricts comprehensiveness if relevant studies are not indexed there and, in addition, complicates comparison with traditional methodologies that rely on multiple bibliographic sources. Regarding article screening, Elicit showed several advantages, including the rapid identification of relevant literature, the structured organization of extracted information (columns based on predefined criteria), and the reasoning supporting the relevance score supported by the corresponding text excerpts. The two-step workflow which combines abstract screening with extraction of information from full-text articles, produced a final set of articles comparable in both size and composition to the human selection, while successfully including all studies identified by human reviewers. Overall, even when considering a workflow that maintains human involvement (human-in-the-loop), the workload for reviewers is reduced. In our specific experience, for example, the reviewer’s task was limited to examining approximately 70 articles in greater depth, and only 24 of these in detail. To our knowledge, this is one of the first studies applying the Elicit’s Systematic Reviews workflow in a step-by-step comparison with the traditional umbrella review process and to outline, at the current stage of its development, potential and limitations. Previously, Bernard et al. [23] conducted a similar comparison using Elicit to search and screen papers. On that occasion, the traditional process retrieved 17 articles, while Elicit retrieved 6 articles, of which 3 (17.6%) were common. Full-text assessment showed that the three articles identified exclusively by Elicit would not have changed the umbrella review’s conclusions. Likewise, even if only the 6 articles retrieved by Elicit had been considered, the conclusions would have remained unchanged. However, the smaller evidence base could have reduced the level of precision. Another study [36] used the Elicit Systematic Review tool in a comparison of literature searches across four case study evidence syntheses, which included two systematic reviews, one umbrella review, and one scoping review in the fields of public health, pharmacology, and surgical procedures. Elicit’s showed poor sensitivity, averaging 39.5% (range 25.5–69.2%), compared to 94.5% (range 91.1–98.0%) in the original reviews. On the other hand, it had high precision (41.8%, range 35.6–46.2%) compared to the original reviews (7.55%, range 0.65–14.7%). Elicit was able to identify some studies that were not found in the original searches, which, interestingly, were notified to the authors of the original reviews to assess whether they met the original inclusion criteria, representing unique additional findings. Also these findings support the role of Elicit as a useful supplementary tool for systematic reviews. The standard version of Elicit was compared to GPT-4 turbo and traditional process [38] as a tool to support qualitative data extraction in systematic reviews. Elicit produced results of significantly higher quality than GPT-4 Turbo and closer to those of human reviewers. It retrieved information for almost all questions–article pairs (99.4%), showing high sensitivity but generating many false positives (27.3%) and longer, less selective responses. Although Elicit rarely missed relevant information, its tendency to over-generate text highlights the continued need for human validation. One of the distinctive aspects of our study is that it started from submitting a PECOS question to the LLM-based tool. Similarly, Li et al. [24] developed and validated a human-in-the-loop AI-assisted system for conducting systematic literature reviews based entirely on GPT-4 in a zero-shot setting. While in our study the PECOS question was progressively and empirically refined through a fine-tuning, also considering that Elicit does not include any option for prompt engineering, in that case the PICOS question was integrated into GPT-4 prompts, providing structured and transparent inclusion/exclusion criteria. Considering the performance of Elicit in relation to the AMSTAR-2 EH questionnaire, our results indicate that not all items can be handled autonomously by Elicit, rather some of them may require expert interpretation to yield an appropriate answer. For instance, in the paper by Boogaard et al. [39] “Long-term exposure to traffic-related air pollution and selected health outcomes: A systematic review and meta-analysis”), both reviewers answered “Yes” to the AMSTAR2-EH question 6 (“Did the authors report the forest plots?”), while Elicit answered “No.” Elicit reported the item as missing because the full text did not include an actual forest plot graph, although the meta-analysis result was present in tabular form suggesting that the algorithm, different to the reviewer, was unable to identify the substantive requirement. In addition, it should be noted that Elicit is currently unable to read or interpret graphs. This limitation can be significantly detrimental when analyzing a systematic review, where the forest plot represents a crucial source of information. Elicit’s weaknesses emerged for complex AMSTAR-2 EH questions that encompassed multiple sub-questions; breaking these down into simpler components could improve performance. Discrepancies between Elicit’s and reviewers’ assessments frequently occurred in cases where the two reviewers had initially disagreed and consensus was required. Moreover, Elicit was unable to provide answers when relevant information was contained in supplementary materials or cross-referenced papers. Limitations As already discussed, the lack of transparency and the non-repeatability of the tool may be considered limitations. Another possible limitation of the study is related to the tool's inability to handle a large number of articles in its search process. At the time our research was conducted, the tool (in its “Pro” subscription version) allowed users to select up to 500 papers per search, retrieved via its internal search engine and/or uploaded by the user. Additionally, up to 500 more papers could be uploaded, for a total of 1,000 papers per search. In our experience, we chose not to rely on the search generated by Elicit for the Step 3 (Article screening) precisely due to this limitation. Currently, the number of papers that can be selected has been increased to 1,000. However, these numbers are still relatively low and particularly limiting for reviews on broad topics with well-established outcomes, thus representing a factor restricting the use of Elicit. On the other hand, the fact that Elicit ranks the up to 500 articles by plausibility—a key difference compared to traditional databases based on Boolean search—based on natural language interpretation, somewhat mitigates this limitation, as it theoretically reduces the possibility that papers of interest would be positioned lower in the selection list provided by the tool itself. Although the use of LLMs as Elicit is not free from limitations and carries the possibility of error, it should also be acknowledged that human-based screening in systematic reviews is itself prone to error and subjectivity. A methodological review has shown that data extraction errors can occur in up to 63% of cases [40]. Moreover, although high-quality systematic literature reviews rely on multiple independent reviewers applying predefined inclusion and exclusion criteria, subjective judgment still influences decisions and may compromise reproducibility. This challenge is further exacerbated by the heavy workload associated with evaluating large volumes of publications and the need to resolve disagreements among reviewers [22]. In this context, it should also be acknowledged that we did not assess the potential errors that may have occurred during the conduct of the systematic review following standard methodology, and this represents a possible limitation of our study. Future directions Our future research will involve replicating the Elicit-driven process with four additional systematic reviews focusing on health endpoints for which the relationship with air pollution has not yet been well established. This will allow us to further assess and validate the described process. These activities will also take into account ongoing changes introduced by the developers of Elicit. Consequently, beyond process validation, this work may also provide an opportunity to document potential improvements in the tool’s performance. Conclusions The Systematic Reviews workflow of Elicit showed potential in supporting systematic review processes. Noteworthy, it did not exclude any of the reviews selected by expert reviewers, thus suggesting the possibility of reducing the screening burden related to the standard workflow and the volume of material requiring in-depth assessment by expert reviewers. However, several important limitations emerged, including low overlap with the traditional methodology in the search task, reliance on a single database, and the inability to autonomously address several AMSTAR-2 EH items that require expert interpretation. At present, Elicit can complement — but not replace — human judgment, supporting a human-in-the-loop approach for efficient yet reliable evidence synthesis. Supplementary Information Additional file 1. Table S1. Supporting text and tables and reasoning provided by Elicit to justify the articles selection. Additional file 2. Elicit specifications. Specifications of the use of Elicit. Additional file 3. Table A1. Screening criteria used in the screening step on title and abstract. Additional file 4. Table A2. Screening criteria used in the screening step on full-text. Additional file 5. Table A3. AMSTAR-2 EH questions and their reformulation into natural language prompts. Additional file 6. Table A4. Results of the screening on title and abstract on the 324 articles retrieved by the traditional methodology. Additional file 7. Table A5. Score distribution of the articles. Additional file 8. Table A6. Results of the screening on full-text on the 69 articles retrieved by Elicit. Additional file 9. Table A7. Results of the application of AMSTAR-2 EH criteria on the final selection of 15 papers. Additional file 1. Table S1. Supporting text and tables and reasoning provided by Elicit to justify the articles selection. Additional file 2. Elicit specifications. Specifications of the use of Elicit. Additional file 3. Table A1. Screening criteria used in the screening step on title and abstract. Additional file 4. Table A2. Screening criteria used in the screening step on full-text. Additional file 5. Table A3. AMSTAR-2 EH questions and their reformulation into natural language prompts. Additional file 6. Table A4. Results of the screening on title and abstract on the 324 articles retrieved by the traditional methodology. Additional file 7. Table A5. Score distribution of the articles. Additional file 8. Table A6. Results of the screening on full-text on the 69 articles retrieved by Elicit. Additional file 9. Table A7. Results of the application of AMSTAR-2 EH criteria on the final selection of 15 papers.