Psychometric properties of the ORTO-R in an adult population: the Turkish version, reliability, and validity study Authors: Seda Kaya, Esma Asil, Funda Pınar Çakıroğlu, Sümeyye Sertdemir, Yaren Can, Emine Muradoğlu Published: 2024 ABSTRACT PurposeThis study aimed to examine the psychometric properties of a Turkish version of the ORTO-R scale by testing its factorial structure, reliability, and validity in a large adult population.MethodsThis cross-sectional study was conducted between November 2022 and February 2023. Through an online questionnaire, 1077 participants were recruited. Confirmatory factor analysis (CFA) was used to examine the construct validity of the questionnaires. The internal consistency of the ORTO-R scale was evaluated with Cronbach's alpha, Omega and test–retest coefficients. For the convergent and divergent validity of the scale, Pearson correlation analysis was performed on the scores of the Eating Attitude Test-26, the Generalized Anxiety Disorder-7 and the Life Satisfaction Scale.ResultsThe two-factor model fit the data well. Analyses confirmed that the two-factor model of the ORTO-R had acceptable or good fit indices (χ 2/df = 2.126; GFI = 0.997; CFI = 0.992; AGFI = 0.992; TLI = 0.985; RMSEA = 0.032; SRMR = 0.029). According to the reliability coefficients, the orthorexia nervosa and method subscales were found to be reliable. The correlations supported the convergent and divergent validity of the scale.ConclusionsThe results demonstrate that the Turkish version of the ORTO-R is a valid and reliable instrument for assessing orthorexic behaviors in a theoretically meaningful way.Level of evidence: Level V, descriptive study.Supplementary InformationThe online version contains supplementary material available at 10.1007/s40519-024-01705-0. Introduction The term orthorexia nervosa (ON) was first defined by Bratman in the late 1990s as pathological adherence to healthy and correct food [1]. In general, it is defined as disordered eating behavior characterized by strict adherence to a healthy diet and appropriate food choices. However, there is no clear consensus on whether ON is a psychological disorder, an eating disorder or a specific disorder [2, 3]. Therefore, ON is not yet defined explicitly in the Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition (DSM-5) [4] or the International Classification of Diseases, Eleventh Revision (ICD-11) [5]. According to the DSM-5, ON can be categorized under the heading “avoidant/restricted food intake disorder” (ARFID) [4]. If ARFID, which has a broad etiological range, is divided into subcategories in the future, ON may also be included in these categories [6]. Academics have determined common criteria for ON. Among the consensus criteria, there is an obsessive focus on healthy eating, developing behaviors to strictly follow healthy dietary rules by avoiding unhealthy and contaminated foods. Furthermore, adherence to a strict diet can lead to social, academic and psychological distress; eating disorders; and medical conditions related to nutritional deficiencies [3]. People with ON avoid sugar, fat, salt, genetically modified products, and ultra-processed foods. They think a lot about menu planning and spend a lot of time on food preparation, purchasing, and cooking [7]. Due to the increasing interest in healthy nutrition obsession and physical fitness in societies, many studies on ON have been conducted. This increased popularity of healthy eating obsession has also led to an alarming increase in the prevalence of ON. Most studies on the prevalence of ON are based on the ORTO-15 developed by Donini et al. [8]. Later, adaptation studies of this scale in different societies were also conducted [9–12]. Alternative screening tools to the ORTO-15 scale, such as the Barcelona Orthorexia Scale (BOS) [13], the Teruel Orthorexia Scale (TOS) [14], and the Orthorexia Nervosa Inventory (ONI) [15] have also been developed. According to the literature, the ORTO-15 has been the most widely used scale because it is the milestone of the studies conducted in this field [16]. However, this scale has several important limitations. First, the original ORTO-15 form and adaptation studies in different populations have unstable structural validity, inconsistent internal consistency, and low reliability [8, 11]. The psychometric test results of the original ORTO-15 and related adaptations are different (one-, two-, and three-factor structures) [12, 17, 18]. Moreover, the prevalence assessment range is approximately 1–80% [9, 19]. This wide range casts doubt on the validity and reliability of the measurement tool. Furthermore, the ORTO-15 was the first scale developed before the application of consensus diagnostic criteria. Finally, since there is no diagnostic criterion accepted by the authorities, the presence of a threshold value for diagnosing ON has been interpreted as "unsafe", and it is emphasized that this should be avoided [20]. For these reasons, Rogaza and Donini updated the revised version of the ORTO-15 (6 items) with a better factor structure. This scale provides a continuous measurement tool for ON instead of a cutoff value. The item locations were changed to avoid method bias. The original 4-point Likert scale was converted to a 5-point Likert scale, and higher scores indicate an increase in orthorectic behaviors [20]. The aim of this study was to translate the ORTO-R scale, which reflects ON thoughts and behaviors in the best way and has the best criterion fit in the studies conducted thus far, into Turkish through the guidelines and to prove the reliability and construct, convergent-divergent, content and face validity of the Turkish version with a comprehensive statistical analysis. Methods Participants and procedure This was a methodological study with a cross-sectional design. The data were collected between November 2022 and February 2023 by the researchers. Ethical approval was obtained from the Ethics Committee of Ankara University (Decision number: 85434274-050.04.04/681117), and the study was conducted by the guidelines of the Declaration of Helsinki. Online informed consent was obtained from all participants who participated in the study. Study data were collected online via Google Forms. Participants were briefed about the study. The participants’ identity information was not questioned, and anonymity was ensured. Participants were invited to work through social media (Instagram, Facebook, Twitter, WhatsApp, and Telegram). Online consent was obtained from the participants, and those who refused consent were excluded. No financial support was given to the participants. The inclusion criteria for the study were as follows: (1) were adults (18 years or older); (2) were fluent in Turkish; (3) were not pregnant/lactating; and (4) had psychiatric or cognitive disorders. In scale adaptation studies, sample size is often determined based on recommended item-to-participant ratios. Common guidelines suggest using a sample size that is 5 to 10 times the number of items on the scale [21, 22]. Recently, it has been recommended to aim for a participant count close to 1000 [23, 24]. This study included 1,077 participants, meeting the recommended sample size for the psychometric analysis of the ORTO-R scale. Measures In the first part of the questionnaire, the participants' sociodemographic information (age, gender, marital status, educational status, etc.) and physical characteristics (height, weight) were collected; in the second part, the ORTO-R scale was used; in the third part, the Eating Attitude Test-26 (EAT-26) was used; in the fourth part, the Generalized Anxiety Disorder-7 (GAD-7) Scale was used; and in the last part, the Life Satisfaction Scale (LSS) was used. For test–retest analysis, the questionnaire was reposted on social media after 4–6 weeks, and 50 participants who provided consent to participate were reincluded in the study. Questions were added to the questionnaire to determine whether these participants had previously participated in the study. Body mass index (BMI) was calculated as body weight (kg) divided by height (meter) squared [25]. The ORTO-R [20], the revised form of the ORTO-15 [8], consists of 6 items. The scale is a 5-point Likert scale (never, rarely, sometimes, very often, and always). The Eating Attitude Test-26 (EAT-26) was developed by revising the Eating Attitude Test-40 to detect disordered eating attitudes [26]. A study of the validity and reliability of the scale in the Turkish context was conducted by Ergüney et al. [27]. The Generalized Anxiety Disorder-7 (GAD-7) Scale was developed by Spitzer et al. [28] to assess generalized anxiety [28]. A study of the validity and reliability of the scale in the Turkish context was conducted by Konkan et al. [29]. The Life Satisfaction Scale (LSS) was developed by Köse et al. [30] in the Turkish population to determine the general life satisfaction of individuals in their living spaces. Linguistic validation First, permission was obtained from the author. Based on Beaton's methodology field, the translation and cross-cultural adaptation process was carried out in 6 stages [31]. First, the English version of the scale was translated into Turkish by two bilingual translators. In the second step, two different translations were evaluated and combined. In the third step, the Turkish version of the scale was translated into English by a sworn translator. In the fourth step, all these forms were evaluated by an expert committee (linguist, nutritionist, academician). All these suggestions and corrections were evaluated in the fifth step, and the final form was created. Data analysis In the statistical analysis part of the study, first, the demographic characteristics of the individuals were obtained via frequency analysis. Statistical analyses were performed at the p < 0.05 significance level. IBM SPSS 27 (IBM Corp., 2020) and R software [32] were used for the analyses. The analysis results were obtained with the psych [33] and lavaan [34] R packages. The flow diagram of the study is presented in Fig. 1.Fig. 1Flow diagram of the study Flow diagram of the study Reliability analyses The internal consistency of the ORTO-R scale was evaluated with Cronbach's alpha, Omega and test–retest coefficients. Validity analyses To test the construct fit, confirmatory factor analysis (CFA) was applied with the diagonal weighted least squares (DWLS) estimation technique on the two subdimensions of the scale. The chi-square statistic divided by degrees of freedom (χ 2/df), goodness-of-fit index (GFI), comparative fit index (CFI), adjusted goodness-of-fit index (AGFI), Tucker‒Lewis index (TLI), root-mean-square error of approximation (RMSEA), and standardized root-mean-square residual (SRMR) indices were used to test the goodness of fit of the model. For the convergent and divergent validity of the scale, Pearson correlation analysis was performed on the scores of the LSS, the GAD-7 and the EAT-26. Content validity First, an expert opinion form was created. The form was sent to 10 bilingual experts in the field. The item-level content validity index (I-CVI) and scale-level content validity index (S-CVI) at the scale level of the ORTO-R were calculated in line with expert opinions. The Davis technique was used to calculate content validity [35]. In the literature, a value greater than 0.78 indicates item-level content validity, and a value greater than 0.80 indicates scale-level content validity [36]. The I-CVI ranged from 0.90 to 1.00, and the S-CVI ranged from 0.97 (Supplementary File 1). Face validity A pilot study was conducted with 30 participants to assess face validity. Participants were asked open-ended questions regarding each survey question. The comprehensibility, ambiguity and clarity of the items were questioned. The time required to fill out the questionnaire and whether individuals had problems reading, understanding and answering it were evaluated. Results Table 1 shows the descriptive analysis findings regarding the demographic characteristics of the participants in the study. A total of 80.5% of the participants were female, and 87.8% were single. Eighty-eight percent of the participants had an undergraduate or graduate education. The mean age of the participants was 24 ± 7 years. The mean BMI was 22.5 ± 4.1. The ORTO-R scores of women were significantly greater than those of men (t = 4.329, p < 0.001). Based on education levels, the ORTO-R overall mean score of a master’s degree or higher was found to be significantly greater than that of primary education and secondary education graduates (F = 3.813 p = 0.010). Participants with moderate physical activity levels had significantly greater mean ORTO-R scores than did those who answered "no or very little exercise" (F = 4.182 p = 0.006).Table 1Demographic characteristics of the participantsVariablesN = 1077Age (years)24 ± 7BMI (kg/m2)22.5 ± 4.1Gender Female867 (80.5%) Male210 (19.5%)Marital status Single946 (87.8%) Married131 (12.2%)Education level Primary education23 (2.1%) Secondary education106 (9.8%) Tertiary education854 (79.3%) Master’s or higher degree94 (8.7%)Smoking status Yes200 (18.6%) No877 (81.4%)Drinking status Yes221 (20.2%) No856 (79.5%)Physical activity status No or little exercise314 (29.1%) Light exercise (walking 1–3 days a week, etc.)477 (44.3%) Moderate exercise (brisk walking 3 or more days a week, etc.)256 (23.8%) Very active exercise (6 or more days of intense exercise per week)30 (2.8%)Categorical data are presented as n (%); numerical data are presented as Mean ± Standard Deviation Table 1Demographic characteristics of the participantsVariablesN = 1077Age (years)24 ± 7BMI (kg/m2)22.5 ± 4.1Gender Female867 (80.5%) Male210 (19.5%)Marital status Single946 (87.8%) Married131 (12.2%)Education level Primary education23 (2.1%) Secondary education106 (9.8%) Tertiary education854 (79.3%) Master’s or higher degree94 (8.7%)Smoking status Yes200 (18.6%) No877 (81.4%)Drinking status Yes221 (20.2%) No856 (79.5%)Physical activity status No or little exercise314 (29.1%) Light exercise (walking 1–3 days a week, etc.)477 (44.3%) Moderate exercise (brisk walking 3 or more days a week, etc.)256 (23.8%) Very active exercise (6 or more days of intense exercise per week)30 (2.8%)Categorical data are presented as n (%); numerical data are presented as Mean ± Standard Deviation Demographic characteristics of the participants Categorical data are presented as n (%); numerical data are presented as Mean ± Standard Deviation Table 2 shows the descriptive statistics and reliability coefficients obtained from the orthorexia nervosa and method subscales of the adapted ORTO-R scale. The reliability analysis revealed that the Cronbach's alpha and McDonald’s omega reliability coefficients of the orthorexia nervosa and method subscales were greater than 0.60. The overall Cronbach's alpha and McDonald’s omega reliability coefficients of the ORTO-R scale are 0.696 and 0.712, respectively. When the items were removed from the method subscale, no significant increase in the reliability coefficient was observed. In addition, the adjusted correlation values of the subscale items of the ORTO-R scale are positive. According to the reliability coefficients, the orthorexia nervosa and method subscales were found to be highly reliable (0.60 < α < 0.80) [37].Table 2Descriptive statistics and reliability analysis findings of the ORTO-R scaleSubscalesItemsMeanSDMedianCorrected total item correlationsα if item deletedα CronbachωOrthorexia nervosaORTOR-12.1291.038Rarely0.4070.6620.6270.643ORTOR-42.2301.240Rarely0.6400.433ORTOR-51.7641.077Never0.6240.450MethodORTOR-23.6191.231Very often0.6150.5250.6590.661ORTOR-33.1761.242Sometimes0.5600.584ORTOR-63.0071.251Sometimes0.5640.580SD Standard Deviation, α Cronbach Cronbach’s alpha, ω McDonald’s omega Table 2Descriptive statistics and reliability analysis findings of the ORTO-R scaleSubscalesItemsMeanSDMedianCorrected total item correlationsα if item deletedα CronbachωOrthorexia nervosaORTOR-12.1291.038Rarely0.4070.6620.6270.643ORTOR-42.2301.240Rarely0.6400.433ORTOR-51.7641.077Never0.6240.450MethodORTOR-23.6191.231Very often0.6150.5250.6590.661ORTOR-33.1761.242Sometimes0.5600.584ORTOR-63.0071.251Sometimes0.5640.580SD Standard Deviation, α Cronbach Cronbach’s alpha, ω McDonald’s omega Descriptive statistics and reliability analysis findings of the ORTO-R scale SD Standard Deviation, α Cronbach Cronbach’s alpha, ω McDonald’s omega Table 3 shows the test–retest findings for the ORTO-R scale. Test–retest correlation coefficients for the orthorexia nervosa subscale and general scale scores are positive and significant. In general, since the test–retest scores of the ORTO-R scale are interrelated and have similar averages at different times, reliability is ensured in terms of test–retest reliability.Table 3Test–retest findings for the ORTO-R scaleTest–retestCorrelation analysisPaired Samples t testrptpOrthorexia nervosa0.4500.0240.7620.022Method0.1740.407−2.0090.454ORTO-R0.4700.0182.4410.056 Table 3Test–retest findings for the ORTO-R scaleTest–retestCorrelation analysisPaired Samples t testrptpOrthorexia nervosa0.4500.0240.7620.022Method0.1740.407−2.0090.454ORTO-R0.4700.0182.4410.056 Test–retest findings for the ORTO-R scale Table 4 shows the standardized factor loadings of the CFA results of the ORTO-R scale. The standardized factor loadings of all the items in the subscales of the ORTO-R are statistically significant and positive (p < 0.05).Table 4Standardized factor loadings of the ORTO-R scale in confirmatory factor analysesSubscalesItemsStandardized factor loadingspOrthorexia nervosaORTOR-10.447 < 0.001ORTOR-40.733 < 0.001ORTOR-50.631 < 0.001MethodORTOR-20.616 < 0.001ORTOR-30.582 < 0.001ORTOR-60.678 < 0.001 Table 4Standardized factor loadings of the ORTO-R scale in confirmatory factor analysesSubscalesItemsStandardized factor loadingspOrthorexia nervosaORTOR-10.447 < 0.001ORTOR-40.733 < 0.001ORTOR-50.631 < 0.001MethodORTOR-20.616 < 0.001ORTOR-30.582 < 0.001ORTOR-60.678 < 0.001 Standardized factor loadings of the ORTO-R scale in confirmatory factor analyses Table 5 shows the values of chi-square statistics, GFI, AGFI, CFI, TLI, NNFI, RMSEA and SRMR from the fit index values of the CFA findings of the ORTO-R scale. The χ 2/df = 2.126 is below 3; the GFI, CFI, AGFI, and TLI values are above 0.95; and the RMSEA and SRMR values are below 0.05. When the values of the model fit index are analyzed in general, the construct validity results of the ORTO-R scale indicate an acceptable/excellent fit [38, 39].Table 5Results of model fit statistics of the confirmatory factor analyseschi-square statistics χ2χ 2/dfGFICFIAGFITLIRMSEASRMR17.0072.1260.9970.9920.9920.9850.0320.029df degrees of freedom, GFI goodness-of-fit index, CFI comparative fit index, AGFI adjusted goodness-of-fit index, TLI Tucker‒Lewis index, RMSEA root-mean-square error of approximation, SRMR standardized root-mean-square residual Table 5Results of model fit statistics of the confirmatory factor analyseschi-square statistics χ2χ 2/dfGFICFIAGFITLIRMSEASRMR17.0072.1260.9970.9920.9920.9850.0320.029df degrees of freedom, GFI goodness-of-fit index, CFI comparative fit index, AGFI adjusted goodness-of-fit index, TLI Tucker‒Lewis index, RMSEA root-mean-square error of approximation, SRMR standardized root-mean-square residual Results of model fit statistics of the confirmatory factor analyses df degrees of freedom, GFI goodness-of-fit index, CFI comparative fit index, AGFI adjusted goodness-of-fit index, TLI Tucker‒Lewis index, RMSEA root-mean-square error of approximation, SRMR standardized root-mean-square residual Table 6 shows the Pearson correlation matrix showing the results of convergent-divergent analyses between ORTO-R scale scores and GAD-7, LSS, and EAT-26 scale scores. The fact that there is an inverse and significant relationship between the method subscale and ORTO-R general scores and the LSS indicates that divergent validity is achieved. In addition, the fact that there is the same directional and significant relationship between the orthorexia nervosa subscale, method subscale and ORTO-R scale scores and the EAT-26 and GAD-7 scores reveals that convergent validity is achieved.Table 6Correlation findings for divergent and convergent validity of the ORTO-R scaleMethodOrthorexia nervosaORTO-RGAD-7LSSEAT-26Method1Orthorexia nervosa0.359**1ORTO-R0.800**0.847**1GAD-70.260**0.074*0.196**1LSS−0.140**0.019−0.067*−0.355**1EAT-260.420**0.132**0.324**0.148**−0.192**1**p < 0.01, *p < 0.05 Table 6Correlation findings for divergent and convergent validity of the ORTO-R scaleMethodOrthorexia nervosaORTO-RGAD-7LSSEAT-26Method1Orthorexia nervosa0.359**1ORTO-R0.800**0.847**1GAD-70.260**0.074*0.196**1LSS−0.140**0.019−0.067*−0.355**1EAT-260.420**0.132**0.324**0.148**−0.192**1**p < 0.01, *p < 0.05 Correlation findings for divergent and convergent validity of the ORTO-R scale **p < 0.01, *p < 0.05 Discussion The aim of this study was to examine the psychometric properties of the Turkish version of the ORTO-R scale in the general population. To our knowledge, this is the first study of the validity and reliability of the ORTO-R in a Turkish adult population. According to the results of the study, the Turkish version of the ORTO-R scale showed good psychometric properties. The results of the analysis showed adequate internal reliability of the ORTO-R scale (α Cronbach = 0.696, ω = 0.712). In the ORTO-R scale by Rogoza and Donini, the omega coefficient was found to be 0.75 [20]. Similar results were found in other validity and reliability studies. The Cronbach's alpha of the Arabic version in Lebanese adults was 0.755 [40], the omega of the Greek version was 0.65 [41], and the omega of the Chinese version was 0.77 [42]. Cronbach's alpha is a measure of internal consistency that indicates how well the items in a scale are related to each other. The alpha coefficient is sensitive to the number of items in a scale. Reliability tends to decrease when there are fewer items because fewer items may not adequately capture the breadth of the construct being measured, reducing internal consistency. This is especially true for scales with fewer than 10 items [43, 44]. A Cronbach’s alpha above 0.6 can be acceptable, especially in exploratory research or contexts with complex constructs and diverse populations [45, 46]. In the literature, the Cronbach's alpha coefficient of the ORTO-R scale has been found to be above 0.6 in validity and reliability studies conducted in different populations. Its being within an acceptable range can be attributed to the low number of items. In all these studies, these findings indicated satisfactory internal consistency. In addition, a test–retest test was conducted in this study, and the fact that the test–retest scores had similar averages at different times verified the stability of the ORTO-R scale. The results of this study confirmed the two-factor structure, similar to the results of Rogaza and Donini [20] and the fit indices showed excellent fit. Good validity levels were also found in the validity and reliability studies of the ORTO-R in different populations [40–42]. Finally, our findings addressed the convergent-divergent validity of the ORTO-R. This convergent validity was measured by assessing the extent to which the ORTO-R might be related to other indicator variables, such as disordered eating attitudes and generalized anxiety disorder. In addition, the correlation coefficients of the ORTO-R and its subdimensions were compared with those of other validity variables, such as eating attitude, life satisfaction, and anxiety. According to the results of the current study, there was a positive correlation between the ORTO-R score and its subdimensions and between eating attitudes and generalized anxiety. In the literature, a similar relationship was found between ON and disordered eating [15, 47]. At the same time, these results confirm that ON is not only a healthy eating obsession but also directly related to disordered eating [48]. In this study, similar results were obtained between anxiety and the ORTO-R, as in the Arabic version of the study [49]. The ORTO-R seems to be characterized by good convergent and divergent validity, which supports our expectations. Strength and limitations This study has several limitations. Although a large sample size was reached in this study, the research sample was not homogeneous in terms of age, education or gender distribution. Weight information was based on participant self-reports, and BMI was calculated based on these self-reports. Since the design of our study is cross-sectional, it is not suitable for inferring causality. Although self-report questionnaires are the most widely used data collection tool, they may contain information bias due to the possibility of misunderstanding questions. Conclusions The present study demonstrated that the Turkish version of the ORTO-R is a valid and reliable instrument for assessing orthorexic behaviors in a theoretically meaningful way. In addition, to our knowledge, this is the first study to evaluate the psychometric properties of the ORTO-R in the Turkish population. The short version of the ORTO-R facilitates the assessment of ON. It can also represent a valuable tool in clinical practice, leading to more accurate assessments and targeted interventions. Easy detection can contribute to early intervention and new measures. What is already known on this subject? Nowadays, there has been an alarming increase in the prevalence of ON due to societies' obsession with healthy eating and increased interest in physical fitness. ORTO-15 is one of the most popular measures of orthorexic thoughts and behaviors. It is the most widely used scale in this field and serves as the cornerstone of many studies. However, numerous studies have highlighted its potential weaknesses and limitations, such as an unstable factorial structure. As a result, the ORTO-15 was revised, and the validity and reliability of the ORTO-R scale were studied. What this study adds? In the present article, the ORTO-R scale was translated into Turkish following established guidelines. The convergent-divergent, content, face validity, and reliability of the scale were confirmed through comprehensive statistical analysis. The Turkish version of the ORTO-R scale also addressed the limitations noted in the literature. To the best of our knowledge, this study is the first to test the Turkish version of the ORTO-R scale. We believe that the current study will provide valuable insights for future ON research. Supplementary Information Below is the link to the electronic supplementary material.Supplementary material 1. Supplementary material 1.