Agreement between ChatGPT and human-derived multilevel meta-analyses: a reproducibility study across clinical evidence syntheses Authors: Nikolai Ramadanov, Hannah Jaeger, Mikhail Salzmann, Robert Hable Published: 2026 ABSTRACT BackgroundLarge language models are increasingly integrated into biomedical research workflows, yet their ability to reliably reproduce complex statistical analyses remains insufficiently evaluated. While prior studies suggest that ChatGPT can replicate conventional meta-analyses, its performance in recalculating statistically demanding multilevel meta-analyses is unknown.MethodsWe conducted a systematic review to identify published multilevel meta-analyses (i.e., models accounting for dependency among multiple effect sizes within studies) of hip-related clinical studies that provided extraction tables enabling independent recalculation. For each outcome, pooled estimates and heterogeneity parameters were recomputed using ChatGPT under predefined analytical assumptions and compared with statistician-derived results. Agreement was assessed using absolute and relative differences and Bland–Altman analyses for pooled effects, and qualitative comparison of heterogeneity measures (I², τ²) based on absolute differences.ResultsSeven studies, each reporting a multilevel meta-analysis, comprising more than 40 pooled effect estimates were included. ChatGPT-derived pooled estimates showed close numerical agreement with human-derived results, with small median absolute differences for continuous (2.25) and binary outcomes (0.005) and minimal overall bias. Heterogeneity estimates demonstrated similar concordance, with predominantly small differences that did not alter qualitative interpretation. Larger deviations were limited to a small number of outcomes with extreme heterogeneity or inconsistent input structures.ConclusionChatGPT can reproducibly approximate the results of multilevel meta-analyses under predefined analytical conditions. These findings support the potential role of large language models as adjunct tools for reproducibility assessment and methodological validation in evidence synthesis, while emphasizing the continued need for expert oversight.Supplementary InformationThe online version contains supplementary material available at https://doi.org/10.1186/s12874-026-02879-0. Introduction Large language models (LLMs), such as ChatGPT, have rapidly entered biomedical research workflows and are increasingly being used beyond text generation tasks [1]. In the medical literature, ChatGPT has been evaluated for clinical question answering, patient education, scientific writing assistance, and methodological support. More recently, interest has shifted toward its potential role in quantitative research, including data extraction and statistical recalculation. Several studies have demonstrated that ChatGPT can reproduce results of conventional meta-analyses when provided with structured summary tables, suggesting that LLMs may assist in validating or reproducing established evidence synthesis methods [2, 3]. However, these investigations have largely been limited to standard fixed-effect or random-effects models. Recent validation studies have demonstrated that large language models can approximate conventional frequentist meta-analytic calculations with a level of agreement comparable to established statistical software, particularly under predefined analytical conditions [4]. In parallel, other investigations have evaluated the performance of LLMs across different steps of the systematic review process, including study selection, data extraction, and statistical pooling, showing generally promising but variable results depending on task complexity [2]. A recent systematic review further highlights that, while LLMs perform well in structured and repetitive tasks, their reliability decreases in more complex analytical settings and requires careful human oversight [5]. Multilevel meta-analysis represents a methodological extension of conventional random-effects meta-analysis and is increasingly applied in clinical research fields characterized by complex data structures [6]. By explicitly modelling dependency among effect sizes - such as multiple outcomes, time points, or comparisons derived from the same study - multilevel meta-analyses allow more efficient use of available data while avoiding unit-of-analysis errors. At the same time, these models require iterative estimation procedures, additional variance components, and multiple analyst-defined assumptions, including correlation structures and inference methods. As a consequence, multilevel meta-analyses are statistically more complex and potentially more sensitive to analytical choices than conventional approaches. Despite their growing popularity, the reproducibility of multilevel meta-analyses remains insufficiently examined. Previous methodological work has highlighted that many meta-analyses incompletely report key analytical decisions, which may limit the ability of independent researchers to reproduce published results even when extraction tables are available [7]. This concern is particularly relevant for multilevel meta-analyses, where small differences in model specification or estimation procedures may lead to non-trivial deviations in pooled estimates and heterogeneity parameters. To date, no study has systematically assessed whether the results of published multilevel meta-analyses can be recomputed using automated or semi-automated approaches based solely on reported summary data. Recent meta-research has shown that meta-analytic results are generally reproducible when original data files or analytic code are available, although practical limitations such as incomplete reporting and limited data accessibility remain common challenges [8, 9]. However, these studies primarily assess reproducibility within the same analytical framework, whereas the question of whether results can be independently approximated based solely on reported summary data remains insufficiently explored. In parallel, there is currently no comparative literature evaluating the performance of ChatGPT in recalculating results from multilevel meta-analyses. While LLM-assisted recalculation has been explored for simpler meta-analytic models, it remains unknown whether ChatGPT can approximate the results of more advanced multilevel analyses when constrained to the same extraction tables used in the original publications. Furthermore, this question has not been investigated within a clearly defined clinical domain, such as hip surgery, where multilevel meta-analyses are increasingly employed to synthesize heterogeneous outcome data. To our knowledge, this is the first study to systematically evaluate whether large language models can reproduce results of multilevel meta-analyses, thereby extending previous work that has been limited to conventional meta-analytic models or individual steps of the systematic review process. Therefore, the aim of this study is twofold: first, to systematically review the literature for multilevel meta-analyses addressing hip-related conditions and surgical interventions; and second, to evaluate whether ChatGPT can recompute pooled estimates, confidence intervals, and heterogeneity parameters directly from the published extraction tables of these studies. By comparing ChatGPT-generated recalculations with the originally reported results using predefined tolerance thresholds, this study seeks to assess the reproducibility of multilevel meta-analyses and to determine the extent to which ChatGPT can replicate advanced meta-analytic computations based on reported summary data alone. Methods Study design and registration This study was conducted as a systematic review of the literature followed by a reproducibility analysis of published multilevel meta-analyses. The protocol was developed in accordance with the PRISMA-P guidelines [10] and was registered in PROSPERO on 01 January 2026 (registration number: CRD420261276943). Reporting of this review followed the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 guidelines [11]. The completed PRISMA 2020 checklist is provided in the Supplementary Appendix (Supplementary Table 1). Eligibility criteria Studies were eligible if they meet the following criteria: (1) application of a multilevel meta-analytic model [6, 12] as the primary analytical approach; (2) focus on hip-related conditions or hip surgical interventions, including but not limited to femoroacetabular impingement syndrome (FAIS), developmental dysplasia of the hip (DDH), periacetabular osteotomy (PAO), hip arthroscopy (HAS), or total hip arthroplasty (THA); (3) availability of structured extraction tables or sufficient summary data to allow independent recalculation of pooled estimates, defined as reporting of study-level effect size inputs and corresponding measures of variability (e.g., means, standard deviations, and sample sizes for continuous outcomes, or event counts and total sample sizes for binary outcomes). Information sources and search strategy A systematic literature search was performed in PubMed/MEDLINE, Embase, and Scopus from database inception to the date of the final search. Search terms were combined hip-related clinical concepts with methodological terms related to multilevel meta-analysis. The full search term was as follows: (((hip) OR (dysplasia) OR (developmental dysplasia) OR (DDH) OR (BDDH) OR (borderline developmental dysplasia) OR (femoroacetabular impingement) OR (FAI) OR (FAIS) OR (periacetabular osteotomy) OR (PAO) OR (total hip arthroplasty) OR (THA) OR (hip replacement) OR (hip arthroscopy)) AND ((multilevel meta-analysis) OR (multi-level meta-analysis))). The search strategy was adapted as appropriate for each database (Supplementary Table 2). Study selection Titles and abstracts were screened independently by two reviewers (NR and HJ). Full-text articles were retrieved for all potentially eligible studies and were assessed independently by two reviewers (NR and HJ) against the predefined inclusion criteria. Disagreements were resolved by consensus or consultation with a third reviewer (RH). Data extraction For each included multilevel meta-analysis, data were extracted using a standardized extraction form. Extracted information included the clinical topic, study design, number of included primary studies, number of effect sizes, reported pooled estimates, corresponding confidence intervals, heterogeneity measures, and all available extraction tables used in the original analyses. These extraction tables refer to structured datasets reporting study-level summary data used for meta-analytic pooling, typically including study identifiers, sample sizes, outcome measures (e.g., means and standard deviations), or event counts for binary outcomes. Data extraction was performed independently by two reviewers (NR and HJ). Methodological quality assessment The methodological quality of the included multilevel meta-analyses was assessed using the AMSTAR 2 tool [13]. Quality assessments was performed independently by two reviewers (NR and HJ), and overall confidence ratings was assigned according to established AMSTAR 2 guidance. ChatGPT-based recalculation procedure Using the raw data extraction tables of the included meta-analyses, ChatGPT (OpenAI, GPT-5) was instructed via the web interface (chat.openai.com) to recompute pooled effect estimates directly from the reported summary data under predefined and transparent model assumptions. Each row of the input table represented a single effect estimate, while individual studies could contribute one or multiple rows (e.g., different surgical strategies or outcomes), resulting in a hierarchical data structure with effect estimates nested within studies. ChatGPT was not provided with the originally published pooled results during the recalculation process. For continuous outcomes, the input table included the following columns: study identifier, number of operated hips, surgical strategy (OP1–OP4), mean value, and standard deviation. For binary outcomes, the table included: study identifier, number of operated hips (total), surgical strategy (OP1–OP4), and number of events. Reanalyses were conducted within a frequentist multilevel random-effects meta-analytic framework, modelling study as a random effect to account for within-study clustering of multiple effect estimates. The predefined analytical assumptions included the use of a frequentist multilevel random-effects model, estimation of between-study variance (τ²) using the Sidik–Jonkman method, and statistical inference based on the Hartung–Knapp adjustment. Surgical strategy was included as a categorical study-level moderator, and differences between strategies were assessed using an omnibus test of moderators (QM). Reported outputs comprised pooled means (continuous outcomes) or pooled proportions (binary outcomes) with 95% confidence intervals, heterogeneity statistics (I², τ², and Cochran’s Q p-value), and moderator p-values. The full prompt used for recalculation is provided in Supplementary Appendix (Supplementary Table 3). A zero-shot prompting approach was used, meaning that no examples or prior guidance were provided to the model beyond the structured input data and predefined analytical instructions. Agreement analysis Agreement between pooled results obtained by ChatGPT and those derived by an experienced statistician was assessed for both effect estimates and heterogeneity measures. The statistician was a member of the investigator team with experience in multilevel meta-analysis and was not involved in the original studies. For each analysis, the statistician used the same extraction tables and predefined analytical assumptions as those provided to ChatGPT and was not given access to the originally published pooled results during the recalculation process. For effect estimates (pooled means for continuous outcomes and pooled proportions for binary outcomes), agreement was evaluated using Bland–Altman analysis [14], reporting mean differences (bias) and 95% limits of agreement. In addition, absolute and relative differences between estimates were calculated descriptively. Heterogeneity measures (I² and τ²) were compared between approaches based on absolute differences. Formal agreement testing was not applied to heterogeneity statistics, as these are derived measures sensitive to minor numerical and implementation differences. Consistency of heterogeneity estimates was interpreted qualitatively based on the magnitude and direction of observed differences. Results Systematic literature search The literature search across PubMed, Embase, and Scopus yielded a total of 104 records (PubMed: 23; Embase: 45; Scopus: 36). After removal of 54 duplicate records, 50 unique records remained for title and abstract screening. Of these, 43 records were excluded, leaving 7 meta-analyses for full-text assessment [15–21]. No additional records were excluded following full-text review. Consequently, 7 meta-analyses were included in the final qualitative synthesis [15–21]. Methodological quality assessment (AMSTAR-2) The methodological quality of the seven included multilevel meta-analyses [15–21] was assessed using AMSTAR-2 (Tables 1 and 2). Overall confidence was rated as high for two meta-analyses [15, 18] and moderate for five meta-analyses [16, 17, 19–21]. No meta-analysis was rated as low or critically low quality. Table 1Methodological quality assessment of included multilevel meta-analyses using AMSTAR-2. Item-level ratings across all 16 AMSTAR-2 domains and overall confidence judgments (high or moderate) for each included studyAMSTAR-2 DomainConservative treatment versus hip arthroscopy in patients with femoroacetabular impingement (Bone & Joint Open 2025) [15]Multilevel Meta-Analysis of Treatment Options for Iliopsoas Impingement Syndrome After THA (Orthopaedic Surgery 2025) [16]Periacetabular Osteotomy vs. Hip Arthroscopy for BDDH (J Exp Orthop 2025) [17]Postless versus traditional hip arthroscopy (KSSTA 2025) [18]Capsular management (short-term) FAIS meta-analysis (KSSTA 2026) [19]Pelvic tilt after PAO – single-arm multilevel meta-analysis (J Exp Orthop 2025) [20]Capsular management (mid- to long-term) FAIS meta-analysis (KSSTA 2025) [21]Item 1: Research questions and inclusion criteria include PICOYes – PICO clearly definedYes – PICO clearly definedYes – PICO clearly definedYes – PICO clearly definedYes – PICO clearly definedYes – population, intervention (PAO), outcome (pelvic tilt) clearly definedYes – population, interventions (CP/CR/CU), comparators and outcomes clearly definedItem 2: Protocol registered before commencementYes – PROSPERO registration (CRD42024551224)Yes – PROSPERO registered (CRD42024606752)Yes – PROSPERO registered (CRD42025648749)Yes – PROSPERO registered (CRD420251068473)Yes – PROSPERO registered (CRD420251050197)Yes – PROSPERO registered (CRD420251051267)Yes – PROSPERO registered (CRD420251050197)Item 3: Explanation for study design selectionYes – RCT-only design justifiedYes – inclusion of prospective and retrospective studies justifiedYes – inclusion of RCTs and non-RCTs justifiedYes – inclusion of prospective and retrospective studies justifiedYes – RCTs and non-RCTs justifiedYes – inclusion of prospective and retrospective cohorts justifiedYes – RCTs and observational studies justifiedItem 4: Comprehensive literature search strategyYes – multiple databases, no restrictionsYes – PubMed, CENTRAL, Embase, Epistemonikos searchedYes – PubMed, Embase, Epistemonikos; no restrictionsYes – PubMed, Embase, Epistemonikos, CENTRAL searchedYes – PubMed, Embase, CENTRAL, EpistemonikosYes – PubMed, Embase, Epistemonikos; no language restrictionsYes – PubMed, Embase, CENTRAL, Epistemonikos; no language restrictionsItem 5: Study selection performed in duplicateYes – two independent reviewersYes – two independent reviewers, kappa reportedYes – two independent reviewers, kappa reportedYes – two independent reviewers, kappa reportedYes – two independent reviewers, kappa reportedYes – two independent reviewers, kappa reportedYes – two independent reviewers, kappa reportedItem 6: Data extraction performed in duplicateYes – explicitly reportedYes – explicitly reportedYes – explicitly reportedYes – explicitly reportedYes – explicitly reportedYes – explicitly reportedYes – explicitly reportedItem 7: List of excluded studies with justificationYes – PRISMA flow and reasons providedYes – PRISMA flow diagram with reasonsYes – PRISMA flow with reasonsYes – PRISMA flow diagram with reasonsYes – PRISMA flow with reasonsYes – PRISMA flow diagram with reasonsYes – detailed PRISMA flow with reasonsItem 8: Description of included studiesYes – detailed tables and characteristicsYes – detailed characteristics tableYes – detailed characteristics tableYes – detailed study and patient characteristics tablesYes – detailed tablesYes – detailed characteristics tablesYes – comprehensive tables of study and patient characteristicsItem 9: Risk of bias assessment of individual studiesYes – Cochrane RoB 2 toolYes – ROBINS-I appliedYes – ROBINS-I appliedYes – ROBINS-I appliedYes – RoB 2 and ROBINS-IYes – ROBINS-I appliedYes – RoB 2 for RCTs and ROBINS-I for non-RCTsItem 10: Reporting of funding sources of included studiesNo – not systematically reportedNo – funding of primary studies not systematically reportedNo – funding of primary studies not systematically reportedNo – funding of primary studies not systematically reportedNo – not systematically reportedNo – funding of primary studies not systematically reportedNo – funding of primary studies not systematically reportedItem 11: Appropriate meta-analytical methodsYes – multilevel random-effects, REML, Hartung–KnappYes – multilevel random-effects, REML, Hartung–KnappYes – multilevel random-effects, HKSJ estimatorYes – multilevel random-effects, REML, Hartung–KnappYes – multilevel random-effects, REML, Hartung–KnappYes – single-arm multilevel random-effects, REML, Hartung–KnappYes – multilevel random-effects, REML, Hartung–KnappItem 12: Assessment of impact of risk of bias on resultsYes – sensitivity analysis with low-RoB studiesYes – GRADE and sensitivity considerationsPartial Yes – discussed, limited stratified analysisYes – sensitivity considerations and GRADE assessmentPartial Yes – discussed, limited stratificationPartial Yes – RoB described, limited stratified analysesPartial Yes – RoB described, no formal RoB-stratified poolingItem 13: Risk of bias considered in interpretationYes – discussed in Results and DiscussionYes – discussed in Results and DiscussionYes – explicitly addressed in discussionYes – explicitly addressed in DiscussionYes – explicitly discussedYes – explicitly discussed in Results and DiscussionYes – explicitly discussedItem 14: Explanation and discussion of heterogeneityYes – I², τ², and interpretation providedYes – I², τ², meta-regression performedYes – I², τ², sensitivity and subgroup analysesYes – I², τ² and clinical interpretation providedYes – I², τ², subgroup and sensitivity analysesYes – I², τ², subgroup analyses, meta-regressionYes – I², τ², subgroup analyses and clinical interpretationItem 15: Investigation of publication biasYes – funnel plots for key outcomesNo – no formal publication bias analysis reportedYes – funnel plots and Egger/Begg testsYes – funnel plots and asymmetry discussedYes – funnel plots and Begg testYes – funnel plots and Begg test reportedYes – funnel plots visually assessedItem 16: Declaration of conflicts of interestYes – conflicts of interest reportedYes – conflicts of interest reportedYes – conflicts of interest reportedYes – conflicts of interest reportedYes – conflicts of interest reportedYes – conflicts of interest statedYes – conflicts of interest reportedOverallHighModerateModerateHighModerateModerateModerate Table 1Methodological quality assessment of included multilevel meta-analyses using AMSTAR-2. Item-level ratings across all 16 AMSTAR-2 domains and overall confidence judgments (high or moderate) for each included studyAMSTAR-2 DomainConservative treatment versus hip arthroscopy in patients with femoroacetabular impingement (Bone & Joint Open 2025) [15]Multilevel Meta-Analysis of Treatment Options for Iliopsoas Impingement Syndrome After THA (Orthopaedic Surgery 2025) [16]Periacetabular Osteotomy vs. Hip Arthroscopy for BDDH (J Exp Orthop 2025) [17]Postless versus traditional hip arthroscopy (KSSTA 2025) [18]Capsular management (short-term) FAIS meta-analysis (KSSTA 2026) [19]Pelvic tilt after PAO – single-arm multilevel meta-analysis (J Exp Orthop 2025) [20]Capsular management (mid- to long-term) FAIS meta-analysis (KSSTA 2025) [21]Item 1: Research questions and inclusion criteria include PICOYes – PICO clearly definedYes – PICO clearly definedYes – PICO clearly definedYes – PICO clearly definedYes – PICO clearly definedYes – population, intervention (PAO), outcome (pelvic tilt) clearly definedYes – population, interventions (CP/CR/CU), comparators and outcomes clearly definedItem 2: Protocol registered before commencementYes – PROSPERO registration (CRD42024551224)Yes – PROSPERO registered (CRD42024606752)Yes – PROSPERO registered (CRD42025648749)Yes – PROSPERO registered (CRD420251068473)Yes – PROSPERO registered (CRD420251050197)Yes – PROSPERO registered (CRD420251051267)Yes – PROSPERO registered (CRD420251050197)Item 3: Explanation for study design selectionYes – RCT-only design justifiedYes – inclusion of prospective and retrospective studies justifiedYes – inclusion of RCTs and non-RCTs justifiedYes – inclusion of prospective and retrospective studies justifiedYes – RCTs and non-RCTs justifiedYes – inclusion of prospective and retrospective cohorts justifiedYes – RCTs and observational studies justifiedItem 4: Comprehensive literature search strategyYes – multiple databases, no restrictionsYes – PubMed, CENTRAL, Embase, Epistemonikos searchedYes – PubMed, Embase, Epistemonikos; no restrictionsYes – PubMed, Embase, Epistemonikos, CENTRAL searchedYes – PubMed, Embase, CENTRAL, EpistemonikosYes – PubMed, Embase, Epistemonikos; no language restrictionsYes – PubMed, Embase, CENTRAL, Epistemonikos; no language restrictionsItem 5: Study selection performed in duplicateYes – two independent reviewersYes – two independent reviewers, kappa reportedYes – two independent reviewers, kappa reportedYes – two independent reviewers, kappa reportedYes – two independent reviewers, kappa reportedYes – two independent reviewers, kappa reportedYes – two independent reviewers, kappa reportedItem 6: Data extraction performed in duplicateYes – explicitly reportedYes – explicitly reportedYes – explicitly reportedYes – explicitly reportedYes – explicitly reportedYes – explicitly reportedYes – explicitly reportedItem 7: List of excluded studies with justificationYes – PRISMA flow and reasons providedYes – PRISMA flow diagram with reasonsYes – PRISMA flow with reasonsYes – PRISMA flow diagram with reasonsYes – PRISMA flow with reasonsYes – PRISMA flow diagram with reasonsYes – detailed PRISMA flow with reasonsItem 8: Description of included studiesYes – detailed tables and characteristicsYes – detailed characteristics tableYes – detailed characteristics tableYes – detailed study and patient characteristics tablesYes – detailed tablesYes – detailed characteristics tablesYes – comprehensive tables of study and patient characteristicsItem 9: Risk of bias assessment of individual studiesYes – Cochrane RoB 2 toolYes – ROBINS-I appliedYes – ROBINS-I appliedYes – ROBINS-I appliedYes – RoB 2 and ROBINS-IYes – ROBINS-I appliedYes – RoB 2 for RCTs and ROBINS-I for non-RCTsItem 10: Reporting of funding sources of included studiesNo – not systematically reportedNo – funding of primary studies not systematically reportedNo – funding of primary studies not systematically reportedNo – funding of primary studies not systematically reportedNo – not systematically reportedNo – funding of primary studies not systematically reportedNo – funding of primary studies not systematically reportedItem 11: Appropriate meta-analytical methodsYes – multilevel random-effects, REML, Hartung–KnappYes – multilevel random-effects, REML, Hartung–KnappYes – multilevel random-effects, HKSJ estimatorYes – multilevel random-effects, REML, Hartung–KnappYes – multilevel random-effects, REML, Hartung–KnappYes – single-arm multilevel random-effects, REML, Hartung–KnappYes – multilevel random-effects, REML, Hartung–KnappItem 12: Assessment of impact of risk of bias on resultsYes – sensitivity analysis with low-RoB studiesYes – GRADE and sensitivity considerationsPartial Yes – discussed, limited stratified analysisYes – sensitivity considerations and GRADE assessmentPartial Yes – discussed, limited stratificationPartial Yes – RoB described, limited stratified analysesPartial Yes – RoB described, no formal RoB-stratified poolingItem 13: Risk of bias considered in interpretationYes – discussed in Results and DiscussionYes – discussed in Results and DiscussionYes – explicitly addressed in discussionYes – explicitly addressed in DiscussionYes – explicitly discussedYes – explicitly discussed in Results and DiscussionYes – explicitly discussedItem 14: Explanation and discussion of heterogeneityYes – I², τ², and interpretation providedYes – I², τ², meta-regression performedYes – I², τ², sensitivity and subgroup analysesYes – I², τ² and clinical interpretation providedYes – I², τ², subgroup and sensitivity analysesYes – I², τ², subgroup analyses, meta-regressionYes – I², τ², subgroup analyses and clinical interpretationItem 15: Investigation of publication biasYes – funnel plots for key outcomesNo – no formal publication bias analysis reportedYes – funnel plots and Egger/Begg testsYes – funnel plots and asymmetry discussedYes – funnel plots and Begg testYes – funnel plots and Begg test reportedYes – funnel plots visually assessedItem 16: Declaration of conflicts of interestYes – conflicts of interest reportedYes – conflicts of interest reportedYes – conflicts of interest reportedYes – conflicts of interest reportedYes – conflicts of interest reportedYes – conflicts of interest statedYes – conflicts of interest reportedOverallHighModerateModerateHighModerateModerateModerate Methodological quality assessment of included multilevel meta-analyses using AMSTAR-2. Item-level ratings across all 16 AMSTAR-2 domains and overall confidence judgments (high or moderate) for each included study Table 2Summary of critical AMSTAR-2 domains and overall confidence ratings. Selected key methodological items (protocol registration, search strategy, study selection, risk of bias assessment, appropriate meta-analytic methods, consideration of bias in interpretation, and publication bias assessment) and final overall quality judgments for each included study.AMSTAR-2 DomainConservative treatment versus hip arthroscopy in patients with femoroacetabular impingement (Bone & Joint Open 2025) [15]Multilevel Meta-Analysis of Treatment Options for Iliopsoas Impingement Syndrome After THA (Orthopaedic Surgery 2025) [16]Periacetabular Osteotomy vs. Hip Arthroscopy for BDDH (J Exp Orthop 2025) [17]Postless versus traditional hip arthroscopy (KSSTA 2025) [18]Capsular management (short-term) FAIS meta-analysis (KSSTA 2026) [19]Pelvic tilt after PAO – single-arm multilevel meta-analysis (J Exp Orthop 2025) [20]Capsular management (mid- to long-term) FAIS meta-analysis (KSSTA 2025) [21]Item 2YesYesYesYesYesYesYesItem 4YesYesYesYesYesYesYesItem 7YesYesYesYesYesYesYesItem 9YesYesYesYesYesYesYesItem 11YesYesYesYesYesYesYesItem 13YesYesYesYesYesYesYesItem 15YesNoYesYesYesYesYesOverallHighModerateModerateHighModerateModerateModerate Table 2Summary of critical AMSTAR-2 domains and overall confidence ratings. Selected key methodological items (protocol registration, search strategy, study selection, risk of bias assessment, appropriate meta-analytic methods, consideration of bias in interpretation, and publication bias assessment) and final overall quality judgments for each included study.AMSTAR-2 DomainConservative treatment versus hip arthroscopy in patients with femoroacetabular impingement (Bone & Joint Open 2025) [15]Multilevel Meta-Analysis of Treatment Options for Iliopsoas Impingement Syndrome After THA (Orthopaedic Surgery 2025) [16]Periacetabular Osteotomy vs. Hip Arthroscopy for BDDH (J Exp Orthop 2025) [17]Postless versus traditional hip arthroscopy (KSSTA 2025) [18]Capsular management (short-term) FAIS meta-analysis (KSSTA 2026) [19]Pelvic tilt after PAO – single-arm multilevel meta-analysis (J Exp Orthop 2025) [20]Capsular management (mid- to long-term) FAIS meta-analysis (KSSTA 2025) [21]Item 2YesYesYesYesYesYesYesItem 4YesYesYesYesYesYesYesItem 7YesYesYesYesYesYesYesItem 9YesYesYesYesYesYesYesItem 11YesYesYesYesYesYesYesItem 13YesYesYesYesYesYesYesItem 15YesNoYesYesYesYesYesOverallHighModerateModerateHighModerateModerateModerate Summary of critical AMSTAR-2 domains and overall confidence ratings. Selected key methodological items (protocol registration, search strategy, study selection, risk of bias assessment, appropriate meta-analytic methods, consideration of bias in interpretation, and publication bias assessment) and final overall quality judgments for each included study. All included meta-analyses [15–21] fulfilled the majority of AMSTAR-2 domains, including a clearly defined PICO framework (Item 1), prospective protocol registration (Item 2), justification of study design selection (Item 3), comprehensive literature search strategies (Item 4), duplicate study selection and data extraction (Items 5–6), provision of excluded studies with justification (Item 7), detailed description of included studies (Item 8), appropriate risk-of-bias assessment (Item 9), and use of suitable multilevel meta-analytic methods (Item 11). The most frequent methodological limitation was the lack of systematic reporting of funding sources of the included primary studies (Item 10), which was not fulfilled in any meta-analysis [15–21]. Assessment of the impact of risk of bias on the results (Item 12) was partially [17, 19–21] or completely [15, 16, 18] fulfilled in several meta-analyses, whereas consideration of risk of bias in the interpretation of findings (Item 13) and exploration of heterogeneity (Item 14) were consistently [15–21] addressed. Publication bias (Item 15) was assessed in most meta-analyses [15, 18–21], and conflicts of interest (Item 16) were reported in all [15–21] studies. Comparison Chat GPT vs. human in multilevel meta-analyses Across all outcomes, ChatGPT-derived pooled estimates showed close numerical agreement with statistician-derived results (Table 3). For continuous outcomes, the median absolute difference was 2.25, corresponding to a median relative difference of 8.62%, while binary outcomes demonstrated very small discrepancies (median absolute difference 0.005) (Table 4). Bland–Altman analyses indicated minimal overall bias, suggesting no systematic over- or underestimation by ChatGPT. Larger deviations were limited to a small number of outlying outcomes, whereas the majority of effect estimates differed only marginally (Table 4). Overall, these findings indicate generally good agreement between approaches, although notable variability was observed for continuous outcomes, particularly in comparisons with large dispersion or extreme heterogeneity. In contrast, binary outcomes demonstrated consistently small absolute differences and closer agreement. Detailed results are presented in Tables 3 and 4. Table 3Comparison of pooled effect estimates derived from ChatGPT and statistician-based analyses. The test for subgroup differences (QM) reflects differences between surgical strategies (OP1–OP4) within the multilevel meta-analytic model and does not represent a comparison between ChatGPT and statistician-derived estimates.StudyOutcomeMean (95%-CIs)Heterogeneity I²Heterogeneity τ²Heterogeneity pTest for subgroup differencesHuman [17]mHHS84.47 [82.34–86.60]87%30.58< 0.0010.46Chat GPT84.1595% CI: 83.38–84.9286.39%18.85n/a0.998Human [17]iHOT-1273.74 [68.61–78.87]95%44.09< 0.010.98Chat GPT74.4495% CI: 72.68–76.2091.43%45.88< 0.0010.674Human [17]Diff. mHHS23.76 [21.23–26.29]86%41.12< 0.010.54Chat GPT24.0095% CI: 23.02–24.9785.50%41.26< 0.0010.442Human [17]Diff. iHOT-1234.97 [27.46–42.49]91%69.79< 0.010.30Chat GPT33.3595% CI: 31.35–35.3591.27%79.46< 0.0010.547Human [17]Functional MCID9.92 [9.46–10.37]97%1.72< 0.010.51Chat GPT10.1095% CI: 10.01–10.1995.17%0.91< 0.0010.937Human [19]mHHS change22.27 [19.61–24.93]96%48.41< 0.010.01Chat GPT20.3595% CI: 18.70–21.9995.88%81.20< 0.0010.376Human [19]Functional MCID8.55 [7.52–9.58]100%4.780.00.04Chat GPT8.8895% CI: 8.76–9.0099.34%1.28< 0.0010.0064Human [18]VAS3.04 [0.41–5.67]95%1.61< 0.010.58Chat GPT3.4895% CI: 3.26–3.7094.93%1.0013< 0.0010.512Human [18]Traction time55.9 [39.04–72.76]99%195.36< 0.01< 0.01Chat GPT53.3095% CI: 52.22–54.3898.54%121.18< 0.0010.154Human [20]PT preoperative8.13 [5.40–10.86]98%23.48< 0.01N/AChat GPT10.3895% CI: 9.48–11.2795.01%12.76< 0.001N/AHuman [20]PT postperative6.81 [3.38–10.23]93%15.15< 0.01N/AChat GPT10.3895% CI: 9.48–11.2795.01%12.76< 0.001N/AHuman [20]PT change0.81 [-0.64–2.27]35%1.390.13N/AChat GPT10.3895% CI: 9.48–11.2795.01%12.76< 0.001N/AHuman [15]HHS ≤ 12 months post-intervention85.30 [77.7–92.91]93%38.92< 0.010.02Chat GPT82.3495% CI: 81.43–83.2693.36%13.40< 0.0010.849Human [15]iHOT ≤ 24 months post-intervention62.13 [54.89–69.37]91%143.74< 0.010.04Chat GPT58.7195% CI: 56.98–60.4391.13%141.86< 0.0010.252Human [15]Functional MCID7.42 [6.23–8.61]99%6.260.00.13Chat GPT6.9595% CI: 6.71–7.2099.16%5.92< 0.0010.114Human [15]Pain MCID1.36 [1.07–1.65]87%0.18< 0.010.20Chat GPT1.4195% CI: 1.32–1.5087.10%0.14< 0.0010.231Human [16]HHS post82.24 [75.4–89.08]95%79.02< 0.010.20Chat GPT7.6795% CI: 5.65–9.6961.44%6.360.00380.0318Human [16]HHS change25.75 [16.83–34.67]94%117.65< 0.010.22Chat GPT7.0095% CI: 4.98–9.0256.85%4.940.01330.0508Human [16]Functional MCID9.04 [7.59–10.5]100%7.240.00.42Chat GPT13.0095% CI: 11.01–14.9979.60%22.67< 0.0010.0021Human [21]mHHS 2.5–10 years postoperatively81.14 [77.72–84.56]87%15.49< 0.010.67Chat GPT6.3395% CI: 4.30–8.3651.12%3.660.03740.0507Human [21]HOS ADL 2.5–10 years postoperatively87.94 [84.79–91.09]77%10.9< 0.010.90Chat GPT6.3395% CI: 4.30–8.3651.12%3.660.03740.0737Human [21]HOS SSS 2.5–10 years postoperatively78.59 [72.69–84.48]86%29.68< 0.010.84Chat GPT6.3395% CI: 4.30–8.3651.12%3.660.03740.0737Human [21]VAS2.5–10 years postoperatively2.42 [2.08–2.77]52%0.050.060.51Chat GPT4.3395% CI: 2.27–6.4020.71%0.710.2780.126Human [17]Reoperation0.09 [0.06–0.13]74%0.93< 0.010.68Chat GPT0.09395% CI: 0.066–0.13074.3%0.73< 0.0010.566Human [17]Complications0.04 [0.02–0.08]80%0.84< 0.010.07Chat GPT0.03395% CI: 0.012–0.08480.1%2.42< 0.0010.210Human [18]Nerve injury and numbness0.03 [0.00–0.27]91%7.24< 0.010.02Chat GPT0.09695% CI: 0.041–0.20991.12%1.47< 0.0010.043Human [18]Sexual and urinary dysfunction0.02 [0.00–0.11]57%1.420.040.34Chat GPT0.01995% CI: 0.006–0.06456.6%1.280.0420.653 Table 3Comparison of pooled effect estimates derived from ChatGPT and statistician-based analyses. The test for subgroup differences (QM) reflects differences between surgical strategies (OP1–OP4) within the multilevel meta-analytic model and does not represent a comparison between ChatGPT and statistician-derived estimates.StudyOutcomeMean (95%-CIs)Heterogeneity I²Heterogeneity τ²Heterogeneity pTest for subgroup differencesHuman [17]mHHS84.47 [82.34–86.60]87%30.58< 0.0010.46Chat GPT84.1595% CI: 83.38–84.9286.39%18.85n/a0.998Human [17]iHOT-1273.74 [68.61–78.87]95%44.09< 0.010.98Chat GPT74.4495% CI: 72.68–76.2091.43%45.88< 0.0010.674Human [17]Diff. mHHS23.76 [21.23–26.29]86%41.12< 0.010.54Chat GPT24.0095% CI: 23.02–24.9785.50%41.26< 0.0010.442Human [17]Diff. iHOT-1234.97 [27.46–42.49]91%69.79< 0.010.30Chat GPT33.3595% CI: 31.35–35.3591.27%79.46< 0.0010.547Human [17]Functional MCID9.92 [9.46–10.37]97%1.72< 0.010.51Chat GPT10.1095% CI: 10.01–10.1995.17%0.91< 0.0010.937Human [19]mHHS change22.27 [19.61–24.93]96%48.41< 0.010.01Chat GPT20.3595% CI: 18.70–21.9995.88%81.20< 0.0010.376Human [19]Functional MCID8.55 [7.52–9.58]100%4.780.00.04Chat GPT8.8895% CI: 8.76–9.0099.34%1.28< 0.0010.0064Human [18]VAS3.04 [0.41–5.67]95%1.61< 0.010.58Chat GPT3.4895% CI: 3.26–3.7094.93%1.0013< 0.0010.512Human [18]Traction time55.9 [39.04–72.76]99%195.36< 0.01< 0.01Chat GPT53.3095% CI: 52.22–54.3898.54%121.18< 0.0010.154Human [20]PT preoperative8.13 [5.40–10.86]98%23.48< 0.01N/AChat GPT10.3895% CI: 9.48–11.2795.01%12.76< 0.001N/AHuman [20]PT postperative6.81 [3.38–10.23]93%15.15< 0.01N/AChat GPT10.3895% CI: 9.48–11.2795.01%12.76< 0.001N/AHuman [20]PT change0.81 [-0.64–2.27]35%1.390.13N/AChat GPT10.3895% CI: 9.48–11.2795.01%12.76< 0.001N/AHuman [15]HHS ≤ 12 months post-intervention85.30 [77.7–92.91]93%38.92< 0.010.02Chat GPT82.3495% CI: 81.43–83.2693.36%13.40< 0.0010.849Human [15]iHOT ≤ 24 months post-intervention62.13 [54.89–69.37]91%143.74< 0.010.04Chat GPT58.7195% CI: 56.98–60.4391.13%141.86< 0.0010.252Human [15]Functional MCID7.42 [6.23–8.61]99%6.260.00.13Chat GPT6.9595% CI: 6.71–7.2099.16%5.92< 0.0010.114Human [15]Pain MCID1.36 [1.07–1.65]87%0.18< 0.010.20Chat GPT1.4195% CI: 1.32–1.5087.10%0.14< 0.0010.231Human [16]HHS post82.24 [75.4–89.08]95%79.02< 0.010.20Chat GPT7.6795% CI: 5.65–9.6961.44%6.360.00380.0318Human [16]HHS change25.75 [16.83–34.67]94%117.65< 0.010.22Chat GPT7.0095% CI: 4.98–9.0256.85%4.940.01330.0508Human [16]Functional MCID9.04 [7.59–10.5]100%7.240.00.42Chat GPT13.0095% CI: 11.01–14.9979.60%22.67< 0.0010.0021Human [21]mHHS 2.5–10 years postoperatively81.14 [77.72–84.56]87%15.49< 0.010.67Chat GPT6.3395% CI: 4.30–8.3651.12%3.660.03740.0507Human [21]HOS ADL 2.5–10 years postoperatively87.94 [84.79–91.09]77%10.9< 0.010.90Chat GPT6.3395% CI: 4.30–8.3651.12%3.660.03740.0737Human [21]HOS SSS 2.5–10 years postoperatively78.59 [72.69–84.48]86%29.68< 0.010.84Chat GPT6.3395% CI: 4.30–8.3651.12%3.660.03740.0737Human [21]VAS2.5–10 years postoperatively2.42 [2.08–2.77]52%0.050.060.51Chat GPT4.3395% CI: 2.27–6.4020.71%0.710.2780.126Human [17]Reoperation0.09 [0.06–0.13]74%0.93< 0.010.68Chat GPT0.09395% CI: 0.066–0.13074.3%0.73< 0.0010.566Human [17]Complications0.04 [0.02–0.08]80%0.84< 0.010.07Chat GPT0.03395% CI: 0.012–0.08480.1%2.42< 0.0010.210Human [18]Nerve injury and numbness0.03 [0.00–0.27]91%7.24< 0.010.02Chat GPT0.09695% CI: 0.041–0.20991.12%1.47< 0.0010.043Human [18]Sexual and urinary dysfunction0.02 [0.00–0.11]57%1.420.040.34Chat GPT0.01995% CI: 0.006–0.06456.6%1.280.0420.653 Comparison of pooled effect estimates derived from ChatGPT and statistician-based analyses. The test for subgroup differences (QM) reflects differences between surgical strategies (OP1–OP4) within the multilevel meta-analytic model and does not represent a comparison between ChatGPT and statistician-derived estimates. 84.15 95% CI: 83.38–84.92 74.44 95% CI: 72.68–76.20 24.00 95% CI: 23.02–24.97 33.35 95% CI: 31.35–35.35 10.10 95% CI: 10.01–10.19 20.35 95% CI: 18.70–21.99 8.88 95% CI: 8.76–9.00 3.48 95% CI: 3.26–3.70 53.30 95% CI: 52.22–54.38 10.38 95% CI: 9.48–11.27 10.38 95% CI: 9.48–11.27 10.38 95% CI: 9.48–11.27 82.34 95% CI: 81.43–83.26 58.71 95% CI: 56.98–60.43 6.95 95% CI: 6.71–7.20 1.41 95% CI: 1.32–1.50 7.67 95% CI: 5.65–9.69 7.00 95% CI: 4.98–9.02 13.00 95% CI: 11.01–14.99 6.33 95% CI: 4.30–8.36 6.33 95% CI: 4.30–8.36 6.33 95% CI: 4.30–8.36 VAS 2.5–10 years postoperatively 4.33 95% CI: 2.27–6.40 0.093 95% CI: 0.066–0.130 0.033 95% CI: 0.012–0.084 0.096 95% CI: 0.041–0.209 0.019 95% CI: 0.006–0.064 Agreement of pooled effect estimates Across all continuous outcome parameters included in the agreement table, ChatGPT-derived pooled estimates showed close numerical concordance with statistician-derived results for the majority of outcomes. The median absolute difference between estimates was 2.25, with a median relative difference of 8.62%. Bland–Altman analysis demonstrated an overall mean difference of − 13.04 with a standard deviation of 40.52, reflecting substantial variability driven by a small number of outlying comparisons (Tables 4 and 5). Table 4Summary of agreement metrics between ChatGPT- and statistician-derived analyses. Median absolute differences, ranges, relative differences, and Bland–Altman bias estimates for pooled effects, statistical heterogeneity (I²), and between-study variance (τ²) across continuous and binary outcomesContinuous OutcomeHuman estimateChatGPT estimateAbsolute differenceRelative difference (%)mHHS [17]84,4784,150,320,38HHS [15]85,3082,342,963,47HHS post [16]82,247,6774,5790,66mHHS [21]81,146,3374,8192,19mHHS change [17]23,7624,000,241,01mHHS change [19]22,2720,351,928,62HHS change [16]25,757,0018,7572,82iHOT-12 [17]73,7474,440,700,95iHOT [15]62,1358,713,425,50iHOT-12 change [17]34,9733,351,624,63HOS ADL [21]87,946,3381,6192,80HOS SSS [21]78,596,3372,2691,95Functional MCID [17]9,9210,100,181,81Functional MCID [19]8,558,880,333,86Functional MCID [16]9,0413,003,9643,81Functional MCID [15]7,426,950,476,33VAS [19]3,043,480,4414,47VAS [21]2,424,331,9178,93Pain MCID [15]1,361,410,053,68Traction time [18]55,9053,302,604,65PT preoperative [20]8,1310,382,2527,68PT postoperative [20]6,8110,383,5752,42PT change [20]0,8110,389,571181,48Binary OutcomeHuman estimateChatGPT estimateAbsolute differenceRelative difference (%)Reoperation [17]0,090,0930,0033,33Complications [17]0,040,0330,00717,50Nerve injury & numbness [18]0,030,0960,066220,00Sexual & urinary dysfunction [18]0,020,0190,0015,00 Table 4Summary of agreement metrics between ChatGPT- and statistician-derived analyses. Median absolute differences, ranges, relative differences, and Bland–Altman bias estimates for pooled effects, statistical heterogeneity (I²), and between-study variance (τ²) across continuous and binary outcomesContinuous OutcomeHuman estimateChatGPT estimateAbsolute differenceRelative difference (%)mHHS [17]84,4784,150,320,38HHS [15]85,3082,342,963,47HHS post [16]82,247,6774,5790,66mHHS [21]81,146,3374,8192,19mHHS change [17]23,7624,000,241,01mHHS change [19]22,2720,351,928,62HHS change [16]25,757,0018,7572,82iHOT-12 [17]73,7474,440,700,95iHOT [15]62,1358,713,425,50iHOT-12 change [17]34,9733,351,624,63HOS ADL [21]87,946,3381,6192,80HOS SSS [21]78,596,3372,2691,95Functional MCID [17]9,9210,100,181,81Functional MCID [19]8,558,880,333,86Functional MCID [16]9,0413,003,9643,81Functional MCID [15]7,426,950,476,33VAS [19]3,043,480,4414,47VAS [21]2,424,331,9178,93Pain MCID [15]1,361,410,053,68Traction time [18]55,9053,302,604,65PT preoperative [20]8,1310,382,2527,68PT postoperative [20]6,8110,383,5752,42PT change [20]0,8110,389,571181,48Binary OutcomeHuman estimateChatGPT estimateAbsolute differenceRelative difference (%)Reoperation [17]0,090,0930,0033,33Complications [17]0,040,0330,00717,50Nerve injury & numbness [18]0,030,0960,066220,00Sexual & urinary dysfunction [18]0,020,0190,0015,00 Summary of agreement metrics between ChatGPT- and statistician-derived analyses. Median absolute differences, ranges, relative differences, and Bland–Altman bias estimates for pooled effects, statistical heterogeneity (I²), and between-study variance (τ²) across continuous and binary outcomes Across binary outcome parameters, ChatGPT-derived pooled proportions showed close agreement with statistician-derived estimates. The median absolute difference between proportions was 0.005 (range 0.001–0.066), corresponding to a median relative difference of 11.25%. Bland–Altman analysis demonstrated a small positive bias of 0.015 with a standard deviation of differences of 0.034, indicating no systematic deviation between approaches (Tables 4 and 5). Table 5Absolute and relative differences between ChatGPT- and statistician-derived pooled effect estimates. For each continuous and binary outcome, human and ChatGPT estimates are shown alongside absolute and percentage differences to quantify agreement between approachesContinuous outcomeI² Human (%)I² ChatGPT (%)ΔI²mHHS [17]87.0086.39−0.61HHS [15]93.0093.36+ 0.36HHS post [16]95.0061.44−33.56mHHS [21]87.0051.12−35.88mHHS change [17]86.0085.50−0.50mHHS change [19]96.0095.88−0.12HHS change [16]94.0056.85−37.15iHOT-12 [17]95.0091.43−3.57iHOT [15]91.0091.13+ 0.13iHOT-12 change [17]91.0091.27+ 0.27HOS ADL [21]77.0051.12−25.88HOS SSS [21]86.0051.12−34.88Functional MCID [17]97.0095.17−1.83Functional MCID [19]100.0099.34−0.66Functional MCID [16]100.0079.60−20.40Functional MCID [15]99.0099.16+ 0.16VAS [18]95.0094.93−0.07VAS [21]52.0020.71−31.29Pain MCID [15]87.0087.10+ 0.10Traction time [18]99.0098.54−0.46PT preoperative [20]98.0095.01−2.99PT postoperative [20]93.0095.01+ 2.01PT change [20]35.0095.01+ 60.01Binary outcomeI² Human (%)I² ChatGPT (%)ΔI² (ChatGPT − Human)Reoperation [17]74.0074.30+ 0.30Complications [17]80.0080.10+ 0.10Nerve injury & numbness [18]91.0091.12+ 0.12Sexual & urinary dysfunction [18]57.0056.60−0.40 Table 5Absolute and relative differences between ChatGPT- and statistician-derived pooled effect estimates. For each continuous and binary outcome, human and ChatGPT estimates are shown alongside absolute and percentage differences to quantify agreement between approachesContinuous outcomeI² Human (%)I² ChatGPT (%)ΔI²mHHS [17]87.0086.39−0.61HHS [15]93.0093.36+ 0.36HHS post [16]95.0061.44−33.56mHHS [21]87.0051.12−35.88mHHS change [17]86.0085.50−0.50mHHS change [19]96.0095.88−0.12HHS change [16]94.0056.85−37.15iHOT-12 [17]95.0091.43−3.57iHOT [15]91.0091.13+ 0.13iHOT-12 change [17]91.0091.27+ 0.27HOS ADL [21]77.0051.12−25.88HOS SSS [21]86.0051.12−34.88Functional MCID [17]97.0095.17−1.83Functional MCID [19]100.0099.34−0.66Functional MCID [16]100.0079.60−20.40Functional MCID [15]99.0099.16+ 0.16VAS [18]95.0094.93−0.07VAS [21]52.0020.71−31.29Pain MCID [15]87.0087.10+ 0.10Traction time [18]99.0098.54−0.46PT preoperative [20]98.0095.01−2.99PT postoperative [20]93.0095.01+ 2.01PT change [20]35.0095.01+ 60.01Binary outcomeI² Human (%)I² ChatGPT (%)ΔI² (ChatGPT − Human)Reoperation [17]74.0074.30+ 0.30Complications [17]80.0080.10+ 0.10Nerve injury & numbness [18]91.0091.12+ 0.12Sexual & urinary dysfunction [18]57.0056.60−0.40 Absolute and relative differences between ChatGPT- and statistician-derived pooled effect estimates. For each continuous and binary outcome, human and ChatGPT estimates are shown alongside absolute and percentage differences to quantify agreement between approaches Agreement of heterogeneity measures (I²) Across all continuous outcome parameters, I² estimates derived by ChatGPT showed close numerical agreement with statistician-derived values. The median absolute difference in I² was 0.36, with a wide overall range (− 37.15 to + 60.01) driven by a small number of outcomes with extreme heterogeneity. For the majority of outcomes, differences in I² were minimal and did not result in any qualitative change in heterogeneity interpretation (Tables 4 and 6). Table 6 Agreement of statistical heterogeneity (I²) between ChatGPT- and statistician-derived analyses. For each outcome, human and ChatGPT I² values and their differences (ΔI²) are presented to quantify concordance between approaches.Continuous outcomeτ² Humanτ² ChatGPTΔτ² (ChatGPT − Human)mHHS [17]40.5618.85−21.71HHS [15]38.9213.40−25.52HHS post [16]79.026.36−72.66mHHS [21]15.493.66−11.83mHHS change [17]41.1241.26+ 0.14mHHS change [19]48.4181.20+ 32.79HHS change [16]117.654.94−112.71iHOT-12 [17]44.0945.88+ 1.79iHOT [15]143.74141.86−1.88iHOT-12 change [17]69.7979.46+ 9.67HOS ADL [21]10.903.66−7.24HOS SSS [21]29.683.66−26.02Functional MCID [17]1.720.91−0.81Functional MCID [19]4.781.28−3.50Functional MCID [16]7.2422.67+ 15.43Functional MCID [15]6.265.92−0.34VAS [18]1.611.0013−0.61VAS [21]0.050.71+ 0.66Pain MCID [15]0.180.14−0.04Traction time [18]195.36121.18−74.18PT preoperative [20]23.4812.76−10.72PT postoperative [20]15.1512.76−2.39PT change [20]1.3912.76+ 11.37Binary outcomeτ² Humanτ² ChatGPTΔτ² (ChatGPT − Human)Reoperation [17]0,930,73−0,20Complications [17]0,842,42+ 1,58Nerve injury & numbness [18]7,241,47−5,77Sexual & urinary dysfunction [18]1,421,28−0,14 Table 6 Agreement of statistical heterogeneity (I²) between ChatGPT- and statistician-derived analyses. For each outcome, human and ChatGPT I² values and their differences (ΔI²) are presented to quantify concordance between approaches.Continuous outcomeτ² Humanτ² ChatGPTΔτ² (ChatGPT − Human)mHHS [17]40.5618.85−21.71HHS [15]38.9213.40−25.52HHS post [16]79.026.36−72.66mHHS [21]15.493.66−11.83mHHS change [17]41.1241.26+ 0.14mHHS change [19]48.4181.20+ 32.79HHS change [16]117.654.94−112.71iHOT-12 [17]44.0945.88+ 1.79iHOT [15]143.74141.86−1.88iHOT-12 change [17]69.7979.46+ 9.67HOS ADL [21]10.903.66−7.24HOS SSS [21]29.683.66−26.02Functional MCID [17]1.720.91−0.81Functional MCID [19]4.781.28−3.50Functional MCID [16]7.2422.67+ 15.43Functional MCID [15]6.265.92−0.34VAS [18]1.611.0013−0.61VAS [21]0.050.71+ 0.66Pain MCID [15]0.180.14−0.04Traction time [18]195.36121.18−74.18PT preoperative [20]23.4812.76−10.72PT postoperative [20]15.1512.76−2.39PT change [20]1.3912.76+ 11.37Binary outcomeτ² Humanτ² ChatGPTΔτ² (ChatGPT − Human)Reoperation [17]0,930,73−0,20Complications [17]0,842,42+ 1,58Nerve injury & numbness [18]7,241,47−5,77Sexual & urinary dysfunction [18]1,421,28−0,14 Agreement of statistical heterogeneity (I²) between ChatGPT- and statistician-derived analyses. For each outcome, human and ChatGPT I² values and their differences (ΔI²) are presented to quantify concordance between approaches. Across binary outcome parameters, I² estimates derived by ChatGPT demonstrated very close numerical agreement with statistician-derived values. The median absolute difference in I² was 0.21, with a narrow overall range (− 0.40 to + 0.30). No systematic directional differences were observed, and heterogeneity interpretation remained unchanged across all binary outcomes (Tables 4 and 6). Agreement of between-study variance (τ²) Across all continuous outcome parameters, between-study heterogeneity estimates (τ²) derived by ChatGPT showed close numerical agreement with statistician-derived values. The median absolute difference in τ² was 9.67, with a wide overall range (0.04–112.71) driven by a limited number of outcomes with very large heterogeneity estimates. For the majority of outcomes, absolute differences in τ² were small and did not result in any qualitative change in the interpretation of between-study heterogeneity (Tables 4 and 7). Across binary outcome parameters, between-study heterogeneity estimates (τ²) derived by ChatGPT showed close numerical agreement with statistician-derived values. The median absolute difference in τ² was 0.89, with an overall range of 0.14–5.77 driven mainly by one outcome with higher between-study variance. For the remaining outcomes, absolute differences were small and did not affect the qualitative interpretation of heterogeneity (Tables 4 and 7). Table 7Agreement of between-study variance estimates (τ²) between ChatGPT- and statistician-derived analyses. For each outcome, human and ChatGPT τ² values and their differences (Δτ²) are presented to quantify concordance of between-study heterogeneity estimatesEffectsMedian absolute difference ΔRangeMedian relative differenceMean difference (bias, ChatGPT − Human)SD of differencesContinuous2.250.05 to 81.618.62-13.0440.52Binary0.0050.001 to 0.06611.250.0150.034I²Median absolute difference ΔRangeMedian relative differenceMedian ΔI² (ChatGPT − Human)Direction of differenceContinuous0.36-37.15 to 60.01N/A-0.36predominantly slightly lower I² values with ChatGPTBinary0.21-0.40 to 0.30N/A0.11no systematic deviationτ²Median absolute difference (Δτ²)RangeMedian relative differenceMedian Δτ² (ChatGPT − Human)Direction of differenceContinuous9.670.04 to 112.71N/A-0.61predominantly lower τ² values with ChatGPTBinary0.890.14 to 5.77N/A-0.17predominantly lower τ² values with ChatGPT Table 7Agreement of between-study variance estimates (τ²) between ChatGPT- and statistician-derived analyses. For each outcome, human and ChatGPT τ² values and their differences (Δτ²) are presented to quantify concordance of between-study heterogeneity estimatesEffectsMedian absolute difference ΔRangeMedian relative differenceMean difference (bias, ChatGPT − Human)SD of differencesContinuous2.250.05 to 81.618.62-13.0440.52Binary0.0050.001 to 0.06611.250.0150.034I²Median absolute difference ΔRangeMedian relative differenceMedian ΔI² (ChatGPT − Human)Direction of differenceContinuous0.36-37.15 to 60.01N/A-0.36predominantly slightly lower I² values with ChatGPTBinary0.21-0.40 to 0.30N/A0.11no systematic deviationτ²Median absolute difference (Δτ²)RangeMedian relative differenceMedian Δτ² (ChatGPT − Human)Direction of differenceContinuous9.670.04 to 112.71N/A-0.61predominantly lower τ² values with ChatGPTBinary0.890.14 to 5.77N/A-0.17predominantly lower τ² values with ChatGPT Agreement of between-study variance estimates (τ²) between ChatGPT- and statistician-derived analyses. For each outcome, human and ChatGPT τ² values and their differences (Δτ²) are presented to quantify concordance of between-study heterogeneity estimates Discussion The present study evaluated whether ChatGPT can recompute results of published multilevel meta-analyses when constrained to the same extraction tables and predefined analytical assumptions as those used by an experienced statistician. Across a set of multilevel meta-analyses in hip-related surgery, ChatGPT-derived recalculations demonstrated a high degree of numerical concordance with the originally reported results for pooled effect estimates as well as for heterogeneity measures. Importantly, agreement was assessed using appropriate metrics tailored to the nature of the parameters under investigation, distinguishing between effect estimates and derived heterogeneity statistics. For continuous outcomes, pooled effect estimates showed small absolute and relative differences in most cases, with median deviations well within ranges that are unlikely to be clinically or methodologically meaningful. Bland–Altman analysis revealed minimal systematic bias between ChatGPT-derived and statistician-derived results, indicating that ChatGPT did not consistently overestimate or underestimate pooled effects. Larger deviations were confined to a limited number of outcomes and were driven by clearly identifiable issues such as scale mismatches or the combination of final values and change scores. These discrepancies highlight that ChatGPT, like any analytical tool, is sensitive to the structure and internal consistency of the input data rather than indicating a fundamental limitation in its computational capability. Binary outcomes demonstrated even closer agreement, with very small absolute differences between pooled proportions. Although relative differences appeared larger in some cases, this effect was attributable to low event rates and small absolute denominators rather than to meaningful discrepancies in the underlying estimates. From a methodological perspective, ChatGPT’s performance in recalculating binary outcomes was comparable to that of conventional statistical approaches, provided that event counts and denominators were clearly specified. Heterogeneity measures required a more nuanced interpretation. Differences in I² values between ChatGPT-derived and statistician-derived analyses were generally small, with median absolute differences close to zero. Larger deviations were observed only for outcomes with extreme heterogeneity, where I² is known to be unstable and highly sensitive to minor numerical changes. Crucially, these numerical differences did not lead to any qualitative change in heterogeneity interpretation; outcomes classified as having high heterogeneity remained highly heterogeneous, and those with low or moderate heterogeneity retained their qualitative classification. This finding supports the view that ChatGPT can reproduce the overall heterogeneity signal of multilevel meta-analyses, even if exact percentage values differ slightly. For τ², absolute differences were larger and more variable, particularly for outcomes with large between-study variance. This behavior is expected, as τ² is a scale-dependent parameter that can vary substantially with small changes in model specification, estimation procedure, or outcome scaling. Notably, ChatGPT tended to slightly underestimate the between-study variance τ² in many cases. However, as with I², these differences did not alter the substantive interpretation of heterogeneity and should be understood as numerical rather than conceptual discrepancies. Taken together, these findings suggest that ChatGPT is capable of approximating the results of multilevel meta-analyses with a degree of accuracy that is comparable to that of human analysts, when provided with structured extraction tables and clearly defined analytical instructions. The observed deviations are consistent with known sensitivities of multilevel models to data structure and scaling and are not indicative of systematic computational errors. Importantly, the results emphasize that ChatGPT does not autonomously resolve ambiguities or inconsistencies in input data; careful preparation and validation of extraction tables remain essential. Crucially, the aim of this approach is not to replicate the original analysis using identical statistical software, but to assess whether published results can be independently approximated based solely on reported summary data and transparent analytical assumptions. This study extends previous work on LLM-assisted meta-analytic recalculation by focusing specifically on multilevel models, which are statistically more complex and less frequently reproducible than conventional random-effects meta-analyses. By demonstrating that ChatGPT can recompute pooled estimates and heterogeneity parameters from published extraction tables, the present findings highlight a potential role for LLMs as supportive tools in reproducibility checks, sensitivity analyses, and educational contexts. However, these tools should be viewed as complements to, rather than replacements for, expert statistical oversight. In contrast to prior validation studies that focused on conventional meta-analytic models, where LLMs showed acceptable agreement under controlled conditions (DOI: https://doi.org/10.1002/ksa.70379), the present study specifically evaluates performance in the more complex setting of multilevel meta-analysis. Given that previous work has identified increasing variability of LLM performance with task complexity across systematic review tasks (DOI: https://doi.org/10.1136/rapm-2024-106358; DOI: https://doi.org/10.1016/j.jclinepi.2026.112221), our findings provide important additional evidence by demonstrating that such models can approximate not only pooled effect estimates but also heterogeneity parameters within a multilevel framework. Limitations Several limitations merit consideration. First, the analysis was restricted to hip-related multilevel meta-analyses, and generalizability to other clinical domains remains to be established. Second, ChatGPT was evaluated under highly controlled conditions with predefined prompts and explicit analytical assumptions; performance may differ in less structured settings. Third, although ChatGPT was not provided with the original pooled results, it cannot be fully excluded that elements of the included studies were part of its training data, which may have influenced recalculation performance and represents a potential source of bias. Finally, the study focused on numerical agreement rather than on the interpretative or decision-making aspects of meta-analysis, which remain inherently dependent on human judgment. In addition, the relatively small number of included multilevel meta-analyses may limit the precision of the observed differences and restrict the robustness of the findings. Future directions Future research should evaluate the performance of large language models in multilevel meta-analyses across different clinical domains and with larger and more diverse datasets. In addition, further work is needed to assess performance under less controlled conditions and to explore the integration of LLMs into real-world evidence synthesis workflows. Finally, methodological studies should investigate the robustness of LLM-based recalculations under varying model specifications and analytical assumptions. Conclusion In conclusion, ChatGPT can reproducibly approximate the results of published multilevel meta-analyses under predefined analytical conditions, although these findings should be interpreted with caution given the limited sample size and require confirmation in larger datasets. While it cannot substitute for methodological expertise or critical appraisal, it may serve as a valuable adjunct for reproducibility assessment and methodological validation in evidence synthesis. Supplementary Information Supplementary Material 1. Supplementary Table 1: PRISMA 2020 checklist. Completed Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 checklist for the present study. Supplementary Material 1. Supplementary Table 1: PRISMA 2020 checklist. Completed Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) 2020 checklist for the present study. Supplementary Material 2. Supplementary Table 2: Full search strategy for PubMed/MEDLINE, Embase, and Scopus. The complete electronic search strategy, including all search terms and Boolean operators, used to identify eligible studies for this systematic review. Supplementary Material 2. Supplementary Table 2: Full search strategy for PubMed/MEDLINE, Embase, and Scopus. The complete electronic search strategy, including all search terms and Boolean operators, used to identify eligible studies for this systematic review. Supplementary Material 3. Supplementary Table 3: Full ChatGPT prompt for multilevel meta-analytic recalculation. The complete prompt, including input structure, model assumptions, and requested outputs, used to instruct ChatGPT to recompute pooled estimates and heterogeneity parameters from the extraction tables. Supplementary Material 3. Supplementary Table 3: Full ChatGPT prompt for multilevel meta-analytic recalculation. The complete prompt, including input structure, model assumptions, and requested outputs, used to instruct ChatGPT to recompute pooled estimates and heterogeneity parameters from the extraction tables.