Validation of the adult asthma epidemiological score: a secondary analysis of the EPI-ASTHMA population-based study Authors: Catarina Laranjeira, Cristina Jácome, Rita Amaral, Filipa Bernardo, Jaime Correia-de-Sousa, Joao A Fonseca Published: 2024 ABSTRACT Abstract AbstractObjectiveThe A2 score is an eight-question patient-reported outcome measure that has been validated for ruling in (score ≥4) and ruling out (score 0–1) asthma. However, this screening tool has been validated in a cohort similar to the derivation cohort used. This study aims to validate the predictive accuracy of the A2 score in a primary care population against general practitioner (GP) clinical assessment and to determine whether the proposed cut-offs are the most appropriate.DesignThis accuracy study is a secondary analysis of the EPI-ASTHMA population-based study.SettingPrimary care centres in Portugal.ParticipantsRandom adult participants answered the A2 score by phone interview.OutcomesThose with an A2 score ≥1 (plus 5% with an A2 score of 0) were invited to a diagnostic visit carried out by a GP to confirm or not a diagnosis of asthma. Diagnostic accuracy was assessed using receiver operating characteristic (ROC) curves.ResultsA total of 1283 participants (median 54 (p25–p75 43–66) years; 60% women) were analysed. The A2 score showed high discriminatory power in identifying asthma, with an area under the ROC curve of 82.9% (95% CI 80.4% to 85.4%). The proposed cut-off ≥4 was the most appropriate to rule in asthma (specificity 83.1%, positive predictive value 62.4%, accuracy 78%). Similarly, the proposed cut-off<2 was the most suitable for excluding asthma (sensitivity 92.7%, negative predictive value 93.7%, accuracy 60.5%).ConclusionsThe A2 score is a useful tool to identify patients with asthma in a primary care population.Trial registration numberNCT0516961.Keywords: Asthma, Epidemiology, Patient Reported Outcome Measures STRENGTHS AND LIMITATIONS OF THIS STUDY This study uses a large sample size recruited from the three most populated regions of Portugal. This study uses a large sample size recruited from the three most populated regions of Portugal. An interview guide was used to standardise the procedures among the interviewers, during the phone call interview. An interview guide was used to standardise the procedures among the interviewers, during the phone call interview. We include only participants from primary care centres, which may limit the extrapolation of A2 performance in other settings. We include only participants from primary care centres, which may limit the extrapolation of A2 performance in other settings. As the A2 score was applied during a phone screening interview, we cannot guarantee that all the participants fully understood the questions of this patient-reported outcome measure, and this may have influenced the results obtained. As the A2 score was applied during a phone screening interview, we cannot guarantee that all the participants fully understood the questions of this patient-reported outcome measure, and this may have influenced the results obtained. Introduction Asthma is a chronic disease characterised by a wide range of respiratory symptoms, such as wheezing, shortness of breath, coughing and chest tightness, and by a variable expiratory airflow limitation, both of which vary in time and intensity.1 This is a growing health problem that affects more than 262 million people worldwide, making it one of the most prevalent chronic diseases, which reflects a severe burden on the healthcare system.2 The prevalence of asthma varies considerably across continents, ranging from 3.4% to 8.3%, with Europe having approximately 5.86% of the population suffering from this disease.3 Differences in asthma prevalence among regions are mainly due to real regional variations but may also result from using different definitions of asthma. Indeed, the definition of asthma has not been standardised for use in epidemiological studies, so each study uses a different questionnaire-based methodology, leading to varying asthma estimates and the inability to make comprehensive comparisons.4 5 Ideally, asthma diagnosis is based on the identification of typical symptoms and supported by the performance of lung function tests, such as spirometry with reversibility test.1 However, this makes the diagnosis more expensive and less accessible, especially in resource-limited regions and in population-based studies. The prevalence of asthma symptoms in epidemiological studies has been mainly measured through written questionnaires.5 Commonly, literature reports the use of questionnaires in multinational epidemiological studies on asthma prevalence in adults, mainly the European Community Respiratory Health Survey (ECRHS).6 The Global Allergy and Asthma European Network also conducted a large multicentre European prevalence study using a questionnaire mostly based on the asthma definitions used in the ECRHS,7 and the World Health Survey provides the most information on asthma prevalence in low-income countries.8 In fact, the WHO Global Alliance against Chronic Respiratory Diseases highlights the importance of the development of simple and affordable diagnostic tools for chronic respiratory diseases, which could be adapted for different realities.9 A systematic review of the diagnostic accuracy of screening tests for chronic obstructive pulmonary disease (COPD) compares the use of the COPD Diagnostic Questionnaire (CDQ) against handheld flow metres.10 Moreover, Martinez et al developed the COPD Assessment in Primary Care to Identify Undiagnosed Respiratory Disease and Exacerbation Risk (CAPTURE) questionnaire to identify subjects who would benefit from further diagnostic investigation.11 In the specific context of asthma, Sá Sousa et al designed and validated a patient-reported outcome measure (PROM) for asthma screening, the Adult Asthma Epidemiological Score (A2 score), a short and easy-to-use questionnaire.12 This was the first self-reported questionnaire to be validated against a GP clinical assessment and diagnostic workup for identifying asthma in adults. Furthermore, it showed the ability to rule in and rule out asthma, meaning that it can be applied in prevalence studies as well as used as a screening tool. The cut-offs suggested—score ≥4 to rule in and scores of 0–1 to rule out—were established based on positive and negative predictive values (PPV and NPV, respectively), which are closely related measurements to the prevalence of asthma, so further testing in epidemiological studies is needed. The A2 score showed high accuracy in a validation cohort extracted from the same population of the derivation cohort.12 However, until now, no validation study applied the A2 score to another population, lacking external validation. Therefore, this study aimed to validate the predictive accuracy of the A2 score against a general practitioner (GP) clinical assessment and to determine whether the proposed cut-offs are the most appropriate in this population. Methods Patient and public involvement Patients or the public were not involved in the design, conduct, reporting or dissemination plans of our research. Study design This accuracy study is a secondary analysis of the EPI-ASTHMA population-based nationwide prevalence study (NCT05169619). Further details regarding this study can be found elsewhere.13 We used data collected between May 2021 and September 2023 from 34 primary care centres in the North, Center and Lisbon Metropolitan Area of Portugal. All the participants provided oral informed consent during the phone interview and later a written informed consent during the clinical assessment visit. This study was reported according to The Standards for Reporting of Diagnostic Accuracy Studies (STARD) guidelines.14 Participants The EPI-ASTHMA study included a random sample of subjects aged ≥18 years who were registered in the primary care National Health Service (NHS) database and provided voluntary consent during an invitation phone call. Those with any specific physical and/or cognitive disabilities that prevented them from cooperating with the study procedures (including lung function tests) and/or understanding/answering the self-reported questionnaires were excluded. This secondary analysis included part of the patients included in the EPI-ASTHMA study as data collection for EPI-ASTHMA study was still ongoing. All subjects diagnosed with asthma from the 34 participating primary care centres were included and those without asthma were randomly selected, in order to have a final sample of ~30% with asthma and ~70% without. This distribution, similar to the used in the A2 score original study, was chosen as it is known that accuracy measurements such as PPV and NPV are highly dependent on prevalence.12 Data collection Participants who fulfilled the eligibility criteria were invited for a phone screening interview performed by a centralised team of experienced interviewers. During the interview, they answered the A2 score.12 This score includes eight questions: about previous physician diagnosis (‘Did a physician confirm you had asthma?’ and ‘Do you still have asthma (previously diagnosed by a physician)?’; about asthma medication intake and asthma symptoms. The resulting score for each patient is the direct sum of all positive answers, ranging from 0 to 8. The original authors suggested that asthma presence could be ruled in for scores of 4 or more (PPV of 93.3%, with 99.2% specificity and 89.4% accuracy) and ruled out for scores of 0 to 1 (NPV of 98.2%, with 93.1% sensitivity and 89.4% accuracy).12 Participants with an A2 score ≥1 were invited to a diagnostic visit undertaken in a mobile outpatient clinic in the following 1–3 weeks. For quality control, ~5% of those with an A2 score of 0 were also invited. The confirmation of an asthma diagnosis was carried out by a GP and was based on clinical history, physical examination, lung function tests (spirometry prebronchodilator and postbronchodilator; fractional exhaled nitric oxide measurement), peripheral blood counts (eosinophil) and PROMs (eg, A2 score, Control of Allergic Rhinitis and Asthma Test).13 15 Diagnosis of asthma followed Global Initiative for Asthma (GINA) recommendations,1 relying primarily on the presence and pattern of respiratory symptoms (wheeze, shortness of breath, chest tightness or cough) and supported by objective lung function findings such variable expiratory airflow limitation and high FeNO levels and other objective collected data (eosinophil). Analysis To describe the characteristics of the participants, mean and SD were used for normally distributed variables, while median and IQR (p25–p75) were used for skewed distributions. As for categorical variables, absolute frequencies, proportions and 95% CI were performed. To compare continuous variables between patients with and without asthma, t-tests for independent samples or Mann-Whitney tests were used depending on the normality of variables. To assess associations between two categorical variables, a χ2 test was performed. Internal consistency of the A2 score was assessed by Cronbach α, which was considered adequate if ≥0.70.16 To evaluate the discriminative power of the A2 score in comparison to the GP asthma diagnosis, receiver operating characteristic (ROC) curve analysis was carried out. Sensitivity, specificity, PPV, NPV and accuracy were used as diagnostic accuracy measures. The two cut-off points were validated by analysing the ROC curve performance, which included calculating the Youden index (sensitivity+specificity−1)17 18 and the combination of PPV, NPV, sensitivity and specificity. In making the selection, we also considered the previous cut-offs suggested12 and the fact that a PPV of at least 50% is reasonable for rule in.19 Statistical analysis was performed using IBM SPSS Statistics, V.29 (IBM Corp, Armonk, New York). A p value of less than 0.05 was considered statistically significant. Results Participants This secondary analysis included 1283 participants (figure 1), with a median age (p25–p75) of 54 (43–66) years old, of which 60% were women (table 1). There were no statistically significant differences between participants with and without asthma regarding age, gender, body mass index, smoking status or geographic region of residence (p>0.05) (table 1). Sample characteristics are shown in table 1. Study flow diagram (n=1283). Table 1Characterisation of the populationAsthma (N=385)No asthma (N=898)Total (N=1283)P valueAge (years), median (p25-p75)52 (41–66)54 (44–66)54 (43–66)0.074*Female, n (%)241 (62.6)527 (58.7)768 (60.0)0.190†BMI (kg/m2), median (p25–p75)27.1 (23.9–30.6)‡26.5 (23.9–30.1)§26.7 (23.9–30.4)0.212*Smoking status, n (%)0.196* Never smoker216 (56.1)¶456 (50.8)**672 (52.4) Current smoker72 (18.7)198 (22.0)270 (21.0) Ex-smoker96 (24.9)241 (26.8)337 (26.3)Region, n (%)0.998† North148 (38.4)345 (38.4)493 (38.4) Center66 (17.1)154 (17.1)220 (17.1) Lisbon Metropolitan Area171 (44.4)399 (44.4)570 (44.4)A2 score, median (p25–p75)5 (3-6)2 (1-3)2 (1-4)<0.001*p25-–p75, percentile 25 to percentile 755; BMI, body mass index; Mann-Witney U test;-square test; 8 missing values;17 missing values; 1 missing values; 3 missing values*Mann-Witney U test.†χ2 test.‡8 missing values.§17 missing values.¶1 missing values.**3 missing values.BMIbody mass index Characterisation of the population p25-–p75, percentile 25 to percentile 755; BMI, body mass index; Mann-Witney U test;-square test; 8 missing values;17 missing values; 1 missing values; 3 missing values Mann-Witney U test. χ2 test. 8 missing values. 17 missing values. 1 missing values. 3 missing values. BMIbody mass index Diagnostic accuracy of the A2 score Participants with asthma had a A2 score median (p25–p75) significantly higher than those without asthma (5 (3–6) vs 2 (1–3), p<0.001) (table 1). Internal consistency of the A2 score was adequate (Cronbach’s α 0.746). The graphic representation of the ability of the A2 score to discriminate between participants with and without asthma is shown in figure 2. As the cut-off point increases, the A2 score becomes more sensitive and less specific, the PPV increases and the NPV decreases (table 2). Thus, the higher the score, the more likely it is to predict the asthma diagnosis, however, the higher the false-positive rate. Receiver operating characteristics (ROC) curve for the A2 score. Table 2Diagnostic accuracy measures and predictive valuesA2 scoreN (%)Sensitivity %(95% CI)Specificity %(95% CI)PPV % (95% CI)NPV % (95% CI)≥11122 (66.0)97.9 (96.0 to 99.1)17.0 (14.6 to 19.7)33.6 (32.9 to 34.3)95.0 (90.5 to 97.5)≥2836 (49.2)92.7 (89.7 to 95.1)46.7 (43.4 to 50.0)42.7 (41.1 to 44.4)93.7 (91.2 to 95.6)≥3591 (34.8)82.0 (77.6 to 85.5)69.3 (66.1 to 72.3)53.3 (50.6 to 56.0)89.9 (87.7 to 91.7)≥4404 (23.8)65.5 (60.5 to 70.2)83.1 (80.5 to 85.5)62.4 (58.5 to 66.1)84.9 (83.0 to 86.6)≥5282 (16.6)54.3 (49.2 to 59.4)91.9 (89.9 to 93.6)74.1 (69.3 to 78.4)82.4 (80.8 to 83.9)≥6188 (11.1)38.7 (33.8 to 43.8)95.7 (94.1 to 96.9)79.3 (73.3 to 84.2)78.5 (77.1 to 79.8)≥7108 (6.4)22.6 (18.5 to 27.1)97.7 (96.5 to 98.6)80.6 (72.3 to 86.8)74.6 (73.6 to 75.7)850 (2.9)10.7 (7.8 to 14.2)99.0 (98.1 to 99.5)82.0 (69.1 to 90.3)72.1 (71.4 to 72.8)A2 scoreAdult Asthma Epidemiological ScoreNPVpredictive negative valuePPVpredictive positive value Diagnostic accuracy measures and predictive values A2 scoreAdult Asthma Epidemiological ScoreNPVpredictive negative valuePPVpredictive positive value The discriminatory capacity of the A2 score, summarised by the area under the ROC curve (AUC), was 82.9% (95% CI 80.4 to 85.4). The predictive power of each cut-off point is shown in table 2. The Youden index is at its highest value when the cut point is set at 3 (sensitivity 82%, specificity 69.3%, NPV 89.9% and PPV 53.3%). This corresponds to the optimal trade-off between sensitivity and specificity. To meet the purpose of our study, a cut-off point of less than 2 positive answers (scores of 0 or 1) was chosen to exclude the presence of asthma. This cut-off point showed a high ability to select individuals who should undergo further diagnostic evaluation, and NPV of 93.7% was obtained, with high sensitivity (92.7%) and an accuracy of 60.5% (table 2). Both cut-offs ≥4 or ≥5 could be appropriate to rule in asthma based on their accuracy in identifying asthma cases. Nevertheless, a cut-off of 4 or higher, with a PPV of 62.4%, and a specificity of 83.1%, was selected as being reasonable accurate in identifying asthma cases (78%) (table 2). Discussion This secondary analysis was the first external validation of the accuracy of the A2 score self-reported questionnaire. The A2 score showed good discriminatory power for asthma diagnosis in a Portuguese primary care population. The proposed cut-offs (scores ≥4 to rule in and scores 0–1 to rule out) were validated in this study population. There is sparse literature on the performance of predictive scores for adult asthma. In a pilot study, the Asthma Screening Questionnaire (ASQ), an asthma screening tool, showed high sensitivity (96%) and specificity (100%) to discriminate between asthma cases and control subjects.20 Accuracy of the A2 score could be interpreted as lower than this ASQ. However, it is important to note that the study conducted by Shin et al was based on a small sample size of 50 participants.20 Additionally, the high accuracy reported may be attributed to the methodology used: the cases were recruited from a clinical setting so they may report more symptoms, while the controls were healthy and asymptomatic subjects, and all confounding comorbid conditions were excluded.20 In contrast, all participants in our study were randomly recruited from primary care centres, better mimicking the performance of a screening tool in clinical practice. Pekkanen et al used the ECRHS definitions to develop a continuous asthma score that can identify individuals for further investigation.21 This method uses the same number of questions as the A2 score questionnaire, mainly based on symptom evaluation. However, the main difference lies in the comparator used: the ECRHS score only compares its results with bronchial hyperreactivity; while the A2 score incorporates a GP clinical assessment that includes clinical history, physical examination, pulmonary function tests, peripheral blood counts, and PROMs. The ECRHS questionnaire was applied to the original A2 score study’s data. The study reported an AUC of 86.8% (95% CI: 82.8 to 90.8%), a sensitivity of 87.2% (95% CI: 80.3 to 92.4%), and a specificity of 98.4% (95% CI: 96.7 to 99.3%).12 Compared with the ECRHS questionnaire, the A2 score showed, in our sample, overlapping discrimination power (AUC 82.9%, 95% CI: 80.4 to 85.4%), higher sensitivity to exclude the presence of asthma (92.7%, 95% CI: 89.7% to 95.1%) and low specificity to identify asthma (83.1%, 95% CI: 80.5% to 85.5%). The accuracy of A2 score is also high when compared with the accuracy of other known COPD screening tools. A systematic review found a pooled sensitivity of 64.5% (95% CI: 59.9 to 68.8%) and specificity of 65.2% (95% CI: 52.9 to 75.8%) for the CDQ.10 In our sample, the cut-off selected to rule out asthma (scores of 0 or 1) had higher sensitivity than that reported for the CDQ (92.7% vs 64.5% respectively), and the cut-off to rule in asthma (scores≥4) had higher specificity (83.1% vs 65.2% respectively).10 The CAPTURE questionnaire had lower discrimination power than the A2 score (AUC of 79.5% vs 82.9% respectively).11 This case-finding questionnaire showed a sensitivity of 95.7% and a specificity of 67.8% in differentiating cases from the control subjects with no COPD.11 Compared with the CAPTURE questionnaire’s diagnostic accuracy, our validation study had slightly lower sensitivity to exclude the presence of the disease (92.7% vs 95.7% respectively) and higher specificity to identify the presence of the disease (83.1% vs 67.8%, respectively).11 To select the optimal cut-off points, a balance between sensitivity and specificity is necessary and should be adapted to meet the specific purposes of the score.22 When conducting prevalence studies, it is more crucial to have a cut point with high specificity rather than sensitivity, as the focus is to rule in asthma with few false positives. However, high sensitivity is preferable when the focus is on identifying patients who are candidates for further diagnostic investigation. Therefore, we believe that a cut point with few missed cases is better suited for use as a screening tool or, in this case, to rule out asthma. For this reason, even though the cut point of 3 corresponds to the highest Youden index, we considered that a cut point of 4 or higher to rule in asthma and a cut point of less than 2 to rule out asthma as the most appropriate in our sample, validating the cut-offs proposed by the authors of this score. Moreover, they reported a specificity of 96.7% (95% CI: 94.6 to 98.2%) and PPV of 85% (95% CI: 76.8 to 90.6%) for the rule in cut-off,12 while in our sample, this cut-off showed lower specificity (83.1%, 95% CI: 80.5 to 85.5%) and PPV (62.4%, 95% CI: 58.5 to 66.1%). According to Price et al, a PPV of at least 50% is reasonable,19 so although the PPV found in our study is lower than that reported by Sá-Sousa et al, it is still very reasonable. For the rule out cut-off, the authors reported a sensitivity of 85.7% (95% CI: 78.6 to 91.2%) and a NPV of 95% (95% CI: 92.5 to 96.6%).12 In our sample, this cut-off point had overlapping sensitivity (92.7%, 95% CI: 89.7 to 95.1%) and NPV (93.7%, 95% CI: 91.2 to 95.6%). The discriminative power (AUC; 95% CI) found is slightly lower than that reported by the authors (82.9%; 80.4–85.4% and 90.4%; 87.0–93.9%, respectively). These differences in the measures of diagnostic accuracy and discriminative power may be attributed to variations in symptom prevalence and asthma severity in the specific settings, as well as differences in sample sizes. In fact, our study has a considerably larger sample size compared with the original A2 score study. This study has strengths and limitations that should be acknowledged. The large sample size recruited from the three most populated regions of the country is an important strength. However, we did not include any participants from the primary care centres of southern Portugal. This study used a sample taken only from primary care, which may limit the extrapolation of results to other settings. In future studies, researchers should validate this score in other settings. Another strength is the fact that we excluded patients with any cognitive disability that would prevent them from understanding or answering the A2 score autonomously. However, as this eligibility screening was made during a phone call interview, we cannot guarantee that all the participants fully understood the questions of the score, and this may have influenced the results. In addition, the A2 score was applied by different healthcare professionals which may also have led to small differences in the administration of the A2 score. To overcome this limitation, an interview guide was used to standardise the procedures among the interviewers. Future studies could compare the reliability of the A2 score applied as an electronic PROM and as a phone interview. Of note, validation against a GP clinical assessment grounded in objective measures and diagnostic tests is also a major strength. This differs from other asthma screening questionnaires, which were only validated against a physician’s diagnosis,23 or based solely on lung functional tests such as spirometry and methacholine challenge test.20 Moreover, the choice of cut-offs was not based solely on PPV and NPV but also on the ROC curve performance and the Youden index, which is a strength of this study compared with other questionnaire validation studies, including the A2 score original study. The advantage of the ROC curve analysis is that since it is based on sensitivity and specificity, it is independent of disease prevalence.18 Conclusions The A2 score is a simple and easily self-administered eight-question case-finding tool that has demonstrated good discriminatory power in a large primary care population of Portugal. In this validation study, the A2 score showed good diagnostic accuracy to be used in epidemiological studies, assessing the prevalence of asthma, as well as a screening tool in clinical settings, to identify individuals who would benefit from further investigation. Future studies are necessary to validate this score in different settings and countries and to adapt the questionnaire for use in other languages and cultural contexts. Footnotes Funding: This study was sponsored and funded by AstraZeneca. Prepublication history for this paper is available online. To view these files, please visit the journal online (https://doi.org/10.1136/bmjopen-2024-086493). Patient consent for publication: Consent obtained directly from patient(s). Ethics approval: The study was approved by the ethics committees of the Regional Health Administration of North (CE/2022/117), Center (27/2021) and of Lisbon and Tagus Valley (2775/CES/2022) and of the Local Health Units of Matosinhos (38/CES/JAS) and Alto Minho (38/2021). Participants gave informed consent to participate in the study before taking part. Provenance and peer review: Not commissioned; externally peer-reviewed. Patient and public involvement: Patients and/or the public were not involved in the design, or conduct, or reporting, or dissemination plans of this research. Contributor Information Catarina Laranjeira, Email: catarina_laranjeira2@hotmail.com. Cristina Jácome, Email: cjacome@med.up.pt. Rita Amaral, Email: ritasamaral@gmail.com. Filipa Bernardo, Email: filipa.bernardo@astrazeneca.com. Jaime Correia-de-Sousa, Email: jaimecsousa@gmail.com. Joao A Fonseca, Email: fonseca.ja@gmail.com. Data availability statement Data are available upon reasonable request. References Data Availability Statement Data are available upon reasonable request.