Benchmarking Generative Artificial Intelligence Against Human Judgment in Eating Disorder Case Recognition and Treatment Recommendations Authors: Jake Linardon, Mariel Messer Published: 2026 ABSTRACT ABSTRACTObjectiveGenerative AI is now used to access eating disorder (ED) information, yet the suitability of its outputs remains unclear. Using two vignettes, we evaluated generative AI's ability to accurately identify ED presentations and recommend appropriate care, while benchmarking its performance against published responses from human samples.MethodChatGPT‐5.4 was presented with two vignettes and prompted with questions related to problem identification and treatment recommendation. Vignette one depicted a restrictive ED consistent with anorexia nervosa or atypical anorexia nervosa, with body weight varying across low‐, average‐, and higher‐weight conditions. Vignette two depicted a person with binge‐eating disorder (BED). Responses across 20 prompt administrations per vignette were compared against benchmark data from clinicians for vignette one and community participants for vignette two.ResultsFor vignette one, ChatGPT identified an ED or possible ED in 100% of responses for the low‐ and average‐weight conditions and 90% for the higher‐weight condition, compared with 47%, 21%, and 16% among human clinicians, respectively. It recommended specialized ED treatment 100% of the time across all three weight conditions, compared with 35%, 19%, and 17% of clinicians, respectively. For vignette two, ChatGPT correctly identified BED as the problem in 100% of prompt repetitions (vs. 59% in community participants) and more consistently endorsed evidence‐based treatments than community participants (100% vs. 50%). Supplementary analyses confirmed similar results when prompting Gemini‐3 and Claude‐4.5.ConclusionGenerative AI can accurately identify ED presentations and recommend suitable care in structured scenarios, while showing little evidence of common biases observed in human samples.Keywords: artificial intelligence, diagnosis, eating disorder, large language model, vignettes Introduction Generative artificial intelligence (AI) is increasingly viewed as having the potential to transform the clinical care of eating disorders (EDs) (Linardon and Torous 2025; Norris et al. 2024). Across mental health settings, these tools are already being explored for a range of applications, including forecasting illness course to enable earlier detection of relapse, supporting assessment and diagnosis, automating documentation to reduce clinician burden, and providing on‐demand support beyond conventional service settings (Hua et al. 2025; Torous et al. 2025; Ali et al. 2025). Early evidence points to growing enthusiasm for generative AI among ED clinicians and people experiencing ED symptoms. Survey data indicate that most clinicians and consumers are open to incorporating generative AI tools into ED treatment, citing perceived benefits such as on‐demand support, convenience, and easier access to psychoeducation (Linardon, Liu, et al. 2025). Beyond broad acceptance, initial uptake is already evident: more than half of surveyed clinicians reported using generative AI (mostly ChatGPT), primarily for administrative tasks and as an additional source of information, while one in five consumers reported using it for educational and self‐help purposes (Linardon, Liu, et al. 2025). However, the utility of generative AI tools in ED care will depend on whether their outputs are accurate and clinically appropriate. Outputs that are misleading or clinically misaligned pose risks to both clinicians and patients when used to interpret symptoms, obtain mental health information, or inform treatment decisions, as they could lead to missed illness detection, delayed access to appropriate care, or the reinforcement of unsuitable treatment recommendations (Torous et al. 2025). Evaluations of earlier AI models when tasked with interpreting mental health case vignettes (i.e., insomnia problems, depressive symptoms) raised concerns about incomplete case formulations, poorly tailored therapeutic recommendations, and performance that deteriorated as clinical complexity increased (Dergaa et al. 2024). More recent tests of newer models, however, suggest stronger performance in identifying primary psychiatric diagnoses and recommending appropriate care for some conditions (i.e., suicidality, major depression) (Levkovich and Elyoseph 2023a, 2023b; Smith et al. 2026), with studies also reporting performance comparable to human professionals on selected vignette tasks (Elyoseph and Levkovich 2024; Thotapalli et al. 2025). Despite these advancements, it remains unclear whether the gains in performance of the latest‐generation models extend to EDs, where the heterogeneity of symptom presentations requires a level of diagnostic precision that may challenge current AI capabilities. This study examined the performance of the most recent ChatGPT model on two ED case vignettes for which human benchmark data were available. We evaluated whether ChatGPT could accurately identify ED presentations and clinically appropriate treatment recommendations when presented with symptom profiles reflecting distinct ED diagnoses. We also compared responses with those of the original human samples to determine how its performance aligned with clinician and community judgment under comparable vignette conditions. Method Design We used a comparative vignette‐based design to evaluate ChatGPT's responses to prompts against published human benchmark data. ChatGPT was selected because our prior empirical work across patients, clinicians, and researchers consistently shows that it is the dominant generative AI tool used (> 90%) for psychoeducation, information seeking, and treatment‐ or diagnosis‐related purposes, whereas use of other models is comparatively uncommon (Linardon, Liu, et al. 2025; Linardon, Messer, et al. 2025; Linardon, Thomas, et al. 2025). However, in a series of sensitivity analyses (see below) we also evaluate responses from both Claude 4.5 (Opus) and Gemini 3 as these are also commonly used models. Source Studies and Human Benchmark Data Two published case vignette studies were used as the source of the materials and the human benchmark data. The first study examined whether mental health clinicians' (n = 245) diagnostic impressions and treatment recommendations for a restrictive ED vignette varied according to the person's body weight (Silbiger 2024). Providers were randomized to one of three vignette versions in which the person's symptoms were identical but her weight was described as either below, within, or above the average range for her age and height, permitting comparison of responses related to AN or atypical AN (see Supporting Information). Findings showed that clinicians were more likely to identify the problem as an ED and recommend specialized ED treatment when the person was described as being at a lower body weight (21% and 35%, respectively) compared to average (8% and 19%, respectively) or higher body weight (8% and 18%, respectively), despite identical symptoms (Silbiger 2024). The second study examined mental health literacy for binge‐eating disorder (BED) in a community sample (n = 235; M age = 37.6; 85% female; 96% White) of Canadian adults (Hollett et al. 2023). Participants read a vignette (see Supporting Information) describing a female who presented with symptoms of BED, but did not mention her weight or the term binge eating. Participants completed a mental health literacy survey, including problem recognition, perceived causes, and beliefs about treatment and prognosis. Findings showed that while most participants believed that the person should seek professional help (91%), only 59% correctly identified the problem as BED (Hollett et al. 2023). These source studies were selected because they enabled comparison of ChatGPT's responses to two distinct human benchmarks. The restrictive ED vignette was valuable because it tested whether body weight alone influenced diagnostic and treatment judgments despite identical symptom presentations, allowing examination of whether ChatGPT reproduced similar weight‐related biases. The BED vignette provided a complementary test of whether ChatGPT could recognize a less explicitly labeled ED in a manner comparable to community‐based mental health literacy. Procedure In March 2026, ChatGPT 5.4 was presented with each case vignette and prompted to respond to items used in the source studies. The prompts (see Supporting Information) were submitted to the model on 20 separate occasions to assess the stability of responses across repeated administrations of the same materials. Twenty prompt administrations were used to provide a robust assessment of response consistency, exceeding the number of repetitions commonly used in prior vignette‐based evaluations of large language models. (Levkovich and Elyoseph 2023a; Elyoseph and Levkovich 2024) The high stability of outputs across these iterations suggested that additional administrations were unlikely to yield substantively different findings. Each prompt was submitted in a new session and chat history was cleared prior to submission. No follow‐up clarifications, refinement prompts, or corrective instructions were provided. For Case Vignette 1, the three vignette versions that varied according to weight category were presented in randomized order. Eighty outputs were generated across the two vignette studies. All outputs were transferred verbatim into a spreadsheet for coding and descriptive summarization. Outcomes For Vignette 1, outcomes comprised ChatGPT's responses to items assessing general problem recognition, endorsement of the presence of an ED, endorsement of core restrictive ED symptoms and functional impairment, treatment recommendations, recommended treatment urgency, recommended treatment dose, and likelihood of recommending medical follow‐up. For the open‐ended problem recognition item, responses were coded using the classification framework reported in Silbiger: (Silbiger 2024) “eating disorder,” “possible eating disorder,” “at risk for future eating disorder,” “disordered eating,” or “missed diagnosis,” ordered from most to least severe. Where a response fit multiple categories, it was assigned to the more severe category. ChatGPT responses were coded as “eating disorder” when the model explicitly stated that the person likely or probably had an ED. Responses were coded as “possible eating disorder” when the model described precise diagnostic criteria or symptoms consistent with AN or atypical AN, as well as identifying impairment, but did not explicitly use the term ED. For the free‐response treatment recommendation item, responses were coded using the categories reported in Silbiger, ordered from most to least intensive: “specialized eating disorder treatment,” “therapy, medical referral, and dietician referral,” “therapy and either medical referral or dietician referral,” and “therapy alone.” Two researchers coded the open‐ended responses while blinded to condition, and any disagreements were resolved through consensus. For Vignette 2, outcomes comprised ChatGPT's responses to items assessing perceived weight category, problem recognition, perceived problem type, identification of the main problem, perceived primary cause, endorsement of professional help‐seeking, beliefs about the likely helpfulness of person‐based, therapy‐based, and medication‐based interventions, and expected prognosis with treatment. All items were presented with their full set of prespecified response options. Data Analysis Data were analyzed descriptively. For each vignette condition, responses across the 20 prompt administrations were summarized using percentages for categorical outcomes and means and standard deviations for continuous outcomes. These descriptive data were then compared descriptively with the human benchmark data from the source studies. Sensitivity Analyses Because both case vignettes were drawn from previously published studies, it was not possible to rule out the possibility that the original materials had been included in the training data of the evaluated models. To address this concern, we conducted a set of sensitivity analyses using adapted versions of both vignettes in which the wording, phrasing, and sentence structure were modified while preserving the same underlying clinical presentation and task demands. This approach was intended to reduce the likelihood that model responses reflected reproduction of the original published wording, while maintaining sufficient fidelity to the source materials for meaningful comparison. These adapted vignette prompts were entered into ChatGPT‐5.4 on five separate occasions per condition. Moreover, to examine cross‐model performance, the adapted vignette prompts were also entered into Claude 4.5 and Gemini 3, each on five separate occasions per condition. The adapted vignettes are presented in the Supporting Information, and the corresponding results are reported in Tables S1 and S2. Results Vignette 1 ChatGPT identified an ED or a possible ED 100% of the time across the low weight (vs. 47% for humans) and average weight (vs. 21% for humans) case vignette conditions, and identified an ED or a possible ED 90% of the time for the higher weight condition (vs. 16% for humans). These data are presented in Table 1 and Figure 1 (left panel). ChatGPT provided a “strongly agree” rating 100% of the time to the statement “LW has an eating disorder” across all weight category conditions. ChatGPT almost always “strongly agreed” that the patient exhibited core AN symptoms and functional impairment across conditions. Contrary to human clinicians, in which a minority reported recommending specialized ED treatment across low (35%), average (19%), and high (17%) weight conditions, ChatGPT recommended specialized treatment in 100% of cases across all conditions (Figure 1, right panel). It almost always recommended that treatment should begin immediately (zero days) and would “very likely” recommend a medical follow‐up, irrespective of weight category. For the low weight condition, the average number of treatment sessions recommended by ChatGPT was 31.5 (SD = 9.8) compared to 20.0 (SD = 0.0) for the other two weight categories. TABLE 1Human clinician versus AI comparison on case study questions across the three weight category conditions.Case study 1 questionsLow weightAverage weightHigh weightHuman (n = 80)AI (n = 20)Human (n = 85)AI (n = 20)Human (n = 79)AI (n = 20)Identified problem N (%) N (%) N (%) N (%) N (%) N (%) Eating disorder17 (21.3%)20 (100%)7 (8.2%)12 (60%)6 (7.6%)8 (40%)Possible eating disorder21 (26.3%)—11 (12.9%)8 (40%)7 (8.9%)10 (50%)At risk for eating disorder2 (2.5%)—4 (4.7%)—6 (7.6%)2 (10%)Disordered eating10 (12.5%)—18 (21.2%)—18 (22.8%)—Missed diagnosis30 (37.5%)—45 (52.9%)—42 (53.2%)— M (SD) M (SD) M (SD) M (SD) M (SD) M (SD) “LW has an eating disorder” agreement a 4.0 (0.9)5.0 (0.0)3.5 (0.9)5.0 (0.0)3.8 (0.8)5.0 (0.0)Symptom agreementRestriction and weight loss4.2 (0.7)5.0 (0.0)3.6 (1.0)4.0 (0.5)3.8 (0.8)4.8 (0.3)Weight gain and avoidance4.3 (0.8)5.0 (0.0)4.1 (0.8)5.0 (0.0)4.0 (0.8)5.0 (0.0)Body image disturbance4.4 (0.6)5.0 (0.0)4.1 (0.7)5.0 (0.0)4.0 (0.7)4.9 (0.2)Distress or impairment4.2 (0.6)5.0 (0.0)4.2 (0.7)5.0 (0.0)4.1 (0.7)5.0 (0.0)Recommended treatment N (%) N (%) N (%) N (%) N (%) N (%) Specialized eating disorder treatment28 (35.4%)20 (100%)16 (19.3%)20 (100%)14 (17.7%)20 (100%)Therapy, medical referral, and dietician referral10 (12.7%)7 (8.4%)13 (16.5%)Therapy and either medical referral or dietician referral14 (17.7%)12 (14.5%)16 (20.3%)Therapy alone27 (34.2%)48 (57.8%)36 (45.6%)Treatment recommendations M (SD) M (SD) M (SD) M (SD) M (SD) M (SD) When should LW begin treatment (days)?4.9 (4.9)0 (0.0)7.6 (9.8)0.75 (2.14)8.1 (10.8)0 (0.0)Recommended # of treatment sessions31.2 (24.1)31.5 (9.8)26.7 (22.5)20.0 (0.0)25.3 (21.4)20.0 (0.0)Likelihood of recommending medical follow‐up b 4.7 (0.5)5.0 (0.0)4.3 (0.9)5.0 (0.0)4.4 (0.9)5.0 (0.0) a Agreement ratings rated on a 5‐point scale ranging from 1 (strongly disagree) to 5 (strongly agree). b Recommending medical follow‐up rated on a 5‐point scale, ranging from 1 (very unlikely) to 5 (very likely). Human percentages derived from Silbiger (Silbiger 2024). Human clinician versus AI comparison on case study questions across the three weight category conditions. Agreement ratings rated on a 5‐point scale ranging from 1 (strongly disagree) to 5 (strongly agree). Recommending medical follow‐up rated on a 5‐point scale, ranging from 1 (very unlikely) to 5 (very likely). Human percentages derived from Silbiger (Silbiger 2024). Percentage of human clinicians and ChatGPT identifying an eating disorder and recommending specialized eating disorder treatment across low‐, average‐, and high‐weight vignette conditions. Problem identification in the left panel reflects the combined “eating disorder” and “possible eating disorder” categories. Vignette 2 ChatGPT showed complete consistency across all prompt repeats. It responded 100% of the time with: (a) “recommended weight” when asked about likely weight category (vs. 7.8% for humans); (b) “yes” when asked whether the person had a problem (vs. 89%); (c) “mental health” when describing the problem (vs. 51%); (d) “binge‐eating disorder” as the main problem (vs. 58%); (e) “biological, psychological, and social factors” as the primary cause (vs. 4%); (f) “yes” when asked whether the person should seek professional help (vs. 92%); (g) “family doctor” (vs. 32%), “psychologist” (vs. 77%), and “dietician” (vs. 46%) as the most helpful person‐based interventions; (h) “CBT” (vs. 42%), “other psychotherapies” (vs. 50%), and “self‐help manuals” (vs. 25%) as the most helpful therapy‐based interventions; (i) “antidepressants” as the most helpful medication (vs. 29%); and (j) “full recovery, but problems might reoccur” as the likely prognosis (vs. 70%). See Table 2 for these data. TABLE 2Human vs. AI comparison on mental health literacy for a case study describing a person with binge‐eating disorder.Case study 2 questionsHuman (N = 235) (%)AI (N = 20) (%)Likely weight categoryUnderweight2.2%Recommended weight7.8%100%Overweight/obese90.1%Do you think Jane has a problem?Yes88.9%100%No0.9%Unsure10.2%How would you describe Jane's problemPhysical health1.7%Mental health51.1%100%Both physical and mental health42.6%Other3.8%I do not think Jane has a problem0.9%What is Jane's main problemDepression6.4%Anxiety4.7%Anorexia nervosa0.9%Bulimia nervosa3.0%An eating disorder, but not anorexia/bulimia6.8%Binge‐eating disorder58.7%100%Obesity/overweight1.7%Other4.3%Not sure13.6%Primary cause of Jane's problemBiological0.9%Social20.4%Psychological46.8%Biological, social, and psychological3.8%100%Other1.3%Not sure26.8%Should Jane seek professional help?Yes91.9%100%NonrI do not think Jane has a problemnrPerson‐based interventions most helpful (up to 3 selected)Self‐help support48.9%Family doctor32.8%100%Psychologist77.9%100%Dietician46.4%100%Family member or close friend18.7%Therapy‐based interventions most helpful (up to 3 selected)CBT42.6%100%Other psychological therapies50.2%100%Self‐help manual or book25.1%100%Family counseling/therapy38.7%Behavioral weight loss44.7%Medication‐based interventions most helpful (up to 3 selected)Vitamins and minerals28.1%Antidepressants29.4%100%Anti‐anxiety medication24.3%Herbal medications13.6%Not sure35.3%80%None of the above40%Likely prognosis with treatmentFull recovery with no further problems5.1%Full recovery, but problems might reoccur70.6%100%Partial recovery5.5%Not sure18.3%I do not think Jane has a problem0.4% Note: Some response options that received no endorsement ratings from both human and AI were omitted from this Table. Human percentages are derived from Hollett, Pennell (Hollett et al. 2023). Human vs. AI comparison on mental health literacy for a case study describing a person with binge‐eating disorder. Note: Some response options that received no endorsement ratings from both human and AI were omitted from this Table. Human percentages are derived from Hollett, Pennell (Hollett et al. 2023). Sensitivity Analyses Sensitivity analyses based on modified case vignettes produced findings that were broadly unchanged from the main analyses (Tables S1 and S2). In Modified Case Study 1, ChatGPT, Claude, and Gemini continued to classify the restrictive presentation as an ED or possible ED across weight conditions, with high endorsement of core anorexia nervosa symptoms and near‐universal recommendation of specialized ED treatment and medical follow‐up. Some minor differences emerged in symptom ratings and treatment timing and dose, particularly in the higher‐weight condition, but these did not change the substantive conclusions. In Modified Case Study 2, responses were highly consistent across models and repetitions, with the only meaningful variation relating to the inferred weight category. Discussion Generative AI tools demonstrated highly accurate and consistent performance across both vignette studies. For the restrictive ED vignette, it showed no evidence of the weight‐related bias observed among human clinicians, reliably identifying an ED or possible ED across all weight presentations, strongly endorsing core AN symptoms, and always recommending specialized ED treatment. For the BED vignette, ChatGPT consistently identified BED as the main problem. Compared to community participants, it was considerably more likely to endorse evidence‐based treatment options and rightly recognize that BED emerges from the combined influence of biological, psychological, and social factors. These findings suggest that the latest ChatGPT model performed with a high degree of diagnostic and treatment recommendation accuracy, showed minimal biases, and descriptively performed better than both human clinician and community benchmark samples. These findings suggest that latest‐generation generative AI models may now be capable of identifying distinct ED presentations and credible help options with a level of accuracy that, in some structured contexts, exceeds that of relevant human benchmark groups. Crucially, model responses did not reflect some of the inherent biases that can influence clinician diagnostic impressions and treatment decision‐making, with such biases having the potential to result in certain clinical groups (e.g., atypical AN) being overlooked or receiving inequitable care (Byrne et al. 2025). Given that these tools are already being adopted by both clinicians and those with lived experience (Linardon, Liu, et al. 2025), findings provide some reassurance that newer generative AI models may be capable of providing clinically appropriate ED‐related information. However, these results should not be taken as conclusive evidence that generative AI is ready for autonomous use in real‐world ED care, as this study was limited to two relatively simple vignette tasks and the broader evidence base on the safety of generative AI in mental health remains limited (Torous et al. 2025). Nevertheless, findings offer early support for the potential of generative AI to function as an adjunct for psychoeducation, symptom interpretation, and decision support, while also demonstrating the need for more research to define its capabilities, boundaries, and safe applications. There are limitations to this study. First, conclusions were based on two vignette tasks and may not generalize to the complexity and nuance of real‐world ED presentations. Second, performance was evaluated under structured prompting conditions and may differ in more naturalistic use contexts. A third important limitation is that both case vignettes were drawn from previously published studies and may therefore have been included, directly or indirectly, in the model's training data or in closely related online materials. This is a challenge for all studies using generative AI to formulate diagnostic and treatment decisions. Thus, the present findings should be interpreted as evidence of benchmark performance on these vignette tasks rather than as a definitive test of independent clinical reasoning. However, sensitivity analyses using paraphrased versions of both vignettes yielded essentially identical results, reducing concern that the findings were driven solely by reproduction of the original published wording. Despite this, prior exposure to similar materials cannot be excluded. In conclusion, generative AI demonstrated strong accuracy in identifying ED presentations, generating clinically appropriate treatment recommendations, and doing so without clear evidence of the biases observed in the human benchmark samples. These findings suggest that newer generative AI models may have value as adjunctive tools for psychoeducation and treatment decision support in ED care. More rigorous research is needed to clarify their capabilities, limitations, and safe role in real‐world clinical contexts. Author Contributions Mariel Messer: conceptualization, writing – review and editing, methodology. Jake Linardon: conceptualization, methodology, writing – review and editing, writing – original draft, formal analysis. Lived Experience Involvement Statement No specific efforts were undertaken to involve persons with lived experience in the study design or execution, or in the preparation of this manuscript. Funding The authors have nothing to report. Conflicts of Interest Jake Linardon is an Associate Editor of the International Journal of Eating Disorders and author of this article. He was excluded from editorial decision‐making related to the acceptance of this article for publication in the journal. Supporting information Table S1: Modified case study 1 responses from ChatGPT three weight category conditions. Table S2: AI responses on mental health literacy for the modified case study describing a person with binge‐eating disorder. Acknowledgments The authors used ChatGPT to assist with proofreading of the final version of this manuscript. We confirm that all AI‐assisted content was carefully reviewed and edited to ensure accuracy and appropriateness. Open access publishing facilitated by Deakin University, as part of the Wiley ‐ Deakin University agreement via the Council of Australasian University Librarians. Linardon, J. , and Messer M.. 2026. “Benchmarking Generative Artificial Intelligence Against Human Judgment in Eating Disorder Case Recognition and Treatment Recommendations.” International Journal of Eating Disorders 59, no. 9: 2076–2083. 10.1002/eat.70128. Action Editor: Ruth Striegel Weissman Data Availability Statement The data that support the findings of this study are available from the corresponding author upon reasonable request. References Associated Data Supplementary Materials Table S1: Modified case study 1 responses from ChatGPT three weight category conditions. Table S2: AI responses on mental health literacy for the modified case study describing a person with binge‐eating disorder. Data Availability Statement The data that support the findings of this study are available from the corresponding author upon reasonable request.