External validation of a cardiovascular risk model for Omani patients with type 2 diabetes mellitus: a retrospective cohort study Authors: Fatema Al Oraimi, Amani Al Rawahi, Amira Al Harrasi, Sarah Albusafi, Laila Mohammed Al-Manji, Abdul Hakeem Alrawahi, Asma Ali Al Salmani Published: 2023 ABSTRACT ObjectivesTo externally validate a recently developed cardiovascular disease (CVD) risk model for Omanis with type 2 diabetes mellitus (T2DM).DesignRetrospective cohort study.SettingNine primary care centres in Muscat Governorate, Oman.ParticipantsA total of 809 male and female adult Omani patients with T2DM free of CVD at baseline were selected using a systematic random sampling strategy.Outcome measuresData regarding CVD risk factors and outcomes were collected from the patients’ electronic medical records between 29 August 2020 and 2 May 2021. The ability of the model to discriminate CVD risk was assessed by calculating the area under the curve (AUC) of the receiver-operating characteristic curve. Calibration of the model was evaluated using a Hosmer-Lemeshow χ2 test and the Brier score.ResultsThe incidence of CVD events over the 5-year follow-up period was 4.6%, with myocardial infarction being most frequent (48.6%), followed by peripheral arterial disease (27%) and non-fatal stroke (21.6%). A cut-off risk value of 11.8% demonstrated good sensitivity (67.6%) and specificity (66.5%). The area under the curve (AUC) was 0.7 (95% CI 0.60 to 0.78) and the Brier score was 0.01. However, the overall mean predicted risk was greater than the overall observed risk (11.8% vs 4.6%) and the calibration graph showed a relatively significant difference between predicted and observed risk levels in different subgroups.ConclusionsAlthough the model slightly overestimated the CVD risk, it demonstrated good discrimination. Recalibration of the model is required, after which it has the potential to be applied to patients presenting to diabetic care centres elsewhere in Oman. STRENGTHS AND LIMITATIONS OF THIS STUDY All required cardiovascular disease predictors were included in the study. All required cardiovascular disease predictors were included in the study. The sample size of the study was sufficiently powered. The sample size of the study was sufficiently powered. Data were collected by professional, trained staff following a standardised data collection sheet. Data were collected by professional, trained staff following a standardised data collection sheet. The study was limited to T2DM patients attending primary public health centres. The study was limited to T2DM patients attending primary public health centres. Because it was created in a high-incidence environment, the model might exaggerate risk when applied to environments with reduced disease frequency. Because it was created in a high-incidence environment, the model might exaggerate risk when applied to environments with reduced disease frequency. Introduction Cardiovascular disease (CVD) is a leading cause of morbidity and mortality among patients with type 2 diabetes mellitus (T2DM), causing more than half of all deaths among affected individuals.1 2 Patients with T2DM have a 2–6 times increased risk of death from cardiovascular events, with the 10-year risk of cardiovascular-related death increasing by up to 86% for every decade of living with diabetes.3 4 Moreover, the risk of stroke and myocardial infarction (MI) among patients with T2DM is 6.7 and 8 times greater, respectively, than that of the general population.5 6 Every 1% increase in glycated haemoglobin (HbA1c) level increases the risk of stroke, heart disease and death by 10–30%.7 Multiple factors have been found to contribute to the increased risk of CVD in T2DM, including accelerated atherosclerotic plaque formation and CVD progression.2 Other risk factors include disease duration, hypertension (HTN), lipid parameters and lifestyle factors such as smoking.8 The identification of individuals at greater risk of developing CVD is, therefore, a recommended approach to primary prevention in many countries.9 Various risk assessment tools have been developed in order to assess CVD risk in primary care settings, such as the Framingham Heart Study risk prediction algorithm and the Prospective Cardiovascular Münster Study coronary risk score.10 11 Similarly, the WHO/International Society of Hypertension risk prediction charts are a commonly used CVD risk prediction tool in many countries.12 Recently, Alrawahi et al conducted a retrospective cohort study to establish a cardiovascular risk model for Omani patients with T2DM.13 In this study, a sample of 2039 patients with T2DM were selected from primary care settings in Ad Dakhiliyah Governorate. All patients were free of CVD at baseline in 2009–2010 and were followed-up until either: (1) their first cardiovascular event occurred; (2) the patient died or (3) the end of the data collection period in December 2015.13 Various predictors were used to estimate CVD risk, including HTN, dyslipidaemia, high HbA1c levels, albuminuria, obesity, smoking status and family history of CVD; however, smoking status was not reported among the independent factors in the model. The estimated 5-year probability of CVD according to this model was articulated as per the below formula13 : 1 _ 0.9991Exp XiBi where Exp PXiBi=Exp (0.038 age in years+0.052 T2DM duration in years+0.102 HbA1c% +0.201 total cholesterol in mmol/L+0.912 albuminuria (coded 1 if present)+0.166 HTN (coded 1 if present)+0.005 body mass index (BMI) in kg/m2). Furthermore, in order to allow treating physicians to estimate overall CVD risk, outcomes considered in the model included stroke, peripheral arterial disease (PAD) and coronary heart disease (CHD).13 The model was internally validated using the model derivation sample and a separate validation sample, consisting of 1314 and 405 patients with T2DM, respectively. In both the derivation and internal validation samples, the model showed good discrimination, with areas under the receiver-operating characteristic (ROC) curve of 0.73 (95% CI 0.69 to 0.77) and 0.70 (95% CI 0.59 to 0.75), respectively. Sensitivity and specificity were optimal at a cut-off value of 11.8%. Calibration of the model was satisfactory and the actual difference between mean predicted and observed risk in different risk groups ranged from 0.7% to 3.1% and 0.1%4.2% in the derivation and validation samples, respectively.13 Nevertheless, no external validation studies of the model have yet been conducted among T2DM patients in other governorates of Oman, thereby limiting the generalisability of the model outside of the Ad Dakhiliyah region. As such, the current study aimed to externally validate the CVD risk assessment model among a cohort of patients with T2DM in Muscat Governorate. The study involved a validation analysis of the accuracy of the risk assessment tool among the total included sample as well as among smokers and non-smokers separately. Materials and methods This retrospective cohort study was conducted between August 2020 and May 2021 in Muscat Governorate. A multistage cluster sampling strategy was used to select three out of the six wilayats (districts) within Muscat Governorate, including Muttrah, Bawshar and Al-Seeb. Subsequently, three primary health centres were randomly selected from each wilayat out of the 28 health centres in Muscat Governorate that offers primary healthcare services to the local population in their respective catchment areas. Based on the previous validation study, the required sample size for the current study was calculated to be around 758, based on an expected sensitivity of 73% and at a precision of 10% and confidence level of 95%.13 However, a total of 809 patients were included to increase the power of the study. The inclusion criteria included Omani patients with T2DM aged ≥20 years old who were free of CVD (including CHD, stroke and PAD) at baseline, registered at the diabetes clinics of primary healthcare institutions within Muscat Governorate in 2015 and followed-up until June 2020. A systematic random sampling strategy was used to select the intended sample from a list of eligible patients identified from each of the selected centres. Patients who developed CVD during the follow-up period for any reason other than T2DM and relevant risk factors were excluded, as were patients with end-stage kidney disease and liver cirrhosis and those with missing data concerning their baseline risk factors or CVD outcome status. All diagnoses of T2DM were made according to WHO criteria.14 Standardised assessment forms and management procedures were followed according to the diabetes care guidelines of the Ministry of Health in Oman.15 For the purposes of the current study, the year 2015 was chosen as the baseline point, so that the follow-up period to the time of data collection (which was commenced in 2020) would be 5 years in duration. All participants were assessed for the main CVD risk factors at baseline and then reassessed for the outcome after the 5-year follow-up period. A data collection sheet was used to collect all relevant information, including the participants’ demographic data, data related to CVD risk factors at baseline and CVD outcomes. Data were collected by trained staff and sourced from the patients’ electronic medical records at the selected institutions as well as at other tertiary institutions when required, such as the Sultan Qaboos University Hospital and Royal Hospital. The following baseline factors were considered: gender, age, T2DM duration, BMI, HbA1c, HTN, total cholesterol level, albuminuria, smoking status (yes/no/ex-smoker) and smoking duration. All laboratory measurements and definitions were consistent with those used in the initial model derivation study.13 Only patients with complete data were included in the analysis to circumvent the issue of missing data. For the purposes of this study, CVD outcomes were defined as per the definitions used in the model derivation study.13 Briefly, a CVD outcome was defined as the first fatal or non-fatal CHD, stroke or PAD event. A CHD diagnosis was confirmed by ECG and serum troponin testing. However, in some cases, an ECG stress test (treadmill test) and coronary angiography were required to confirm the diagnosis. In addition, a CT scan was used to confirm a stroke event, while one of the following criteria was used to diagnose PAD: intermittent claudication confirmed by angiography; a clinical diagnosis of gangrene or limb amputation due to an ischaemic cause. To ensure that the participants were free from CVD at the beginning of the study, the same diagnostic criteria were applied at baseline. Outcomes were tracked from baseline until June 2020 by reviewing each patient’s clinical notes and diagnoses for all visits attended during the follow-up period. In addition, as the CVD outcomes included fatal events, death certificates were checked to confirm causes of death. Table 1 presents the definitions of CVD outcomes and cut-off values used in the assessment of key risk factors. Table 1Definitions of cardiovascular outcomes and key risk factors used in the present studyVariableDefinition and cut-off pointCVD outcomeThe first fatal or non-fatal recorded event of:Confirmed physician diagnosis of CHD in the form of stable angina, unstable angina or acute MIConfirmed physician diagnosis of ischaemic or haemorrhagic strokeConfirmed physician diagnosis of PAD (ie, ischaemic limp, gangrene or amputation)HTNPhysician diagnosis of HTN (based on SBP of ≥140 mm Hg or DBP of ≥90 mm Hg confirmed in BP chart readings after excluding other causes)Uncontrolled BPSBP of ≥140 mm Hg or DBP of ≥90 mm HgAlbuminuriaPersistent albumin-to-creatinine ratio of ≥2.5 in men and ≥3.5 in women, confirmed at least twice within 3 months or more, after excluding other causesDyslipidaemiaAt least one of the following:Total cholesterol of ≥5.2 mmol/LLDL of ≥2.6 mmol/LHDL of ≤0.9 mmol/L in men and ≤1.3 in womenTG of ≥1.7 mmol/LBP, blood pressure; CHD, coronary heart disease; CVD, cardiovascular disease; DBP, diastolic blood pressure; HDL, high-density lipoprotein; HTN, hypertension; LDL, low-density lipoprotein; MI, myocardial infarction; PAD, peripheral arterial disease; SBP, systolic blood pressure; TG, triglycerides. Definitions of cardiovascular outcomes and key risk factors used in the present study Confirmed physician diagnosis of CHD in the form of stable angina, unstable angina or acute MI Confirmed physician diagnosis of CHD in the form of stable angina, unstable angina or acute MI Confirmed physician diagnosis of ischaemic or haemorrhagic stroke Confirmed physician diagnosis of ischaemic or haemorrhagic stroke Confirmed physician diagnosis of PAD (ie, ischaemic limp, gangrene or amputation) Confirmed physician diagnosis of PAD (ie, ischaemic limp, gangrene or amputation) Total cholesterol of ≥5.2 mmol/L Total cholesterol of ≥5.2 mmol/L LDL of ≥2.6 mmol/L LDL of ≥2.6 mmol/L HDL of ≤0.9 mmol/L in men and ≤1.3 in women HDL of ≤0.9 mmol/L in men and ≤1.3 in women TG of ≥1.7 mmol/L TG of ≥1.7 mmol/L BP, blood pressure; CHD, coronary heart disease; CVD, cardiovascular disease; DBP, diastolic blood pressure; HDL, high-density lipoprotein; HTN, hypertension; LDL, low-density lipoprotein; MI, myocardial infarction; PAD, peripheral arterial disease; SBP, systolic blood pressure; TG, triglycerides. In order to ensure data quality, data revision for coding, entry and cleaning was performed prior to analysis. Data entry was performed using EpiData Entry software (EpiData Association, Odense, Denmark) to reduce entry errors. Data analysis was conducted using SPSS, V.24.0 (IBM, Armonk, New York). Continuous variables were expressed as means and SDs, while categorical variables were presented as total counts and percentages. The ability of the model to discriminate CVD risk was assessed by calculating the AUC of the ROC curve with 95% CI. In addition, sensitivity, specificity, likelihood ratios and positive and negative predictive values were also determined. Furthermore, the discrimination ability of the model was calculated for smokers separately by calculating the AUC of the ROC curve in order to judge the ability of the model to be applicable to smokers and non-smokers. For calibration, the sample was divided into five groups (quintiles) by sorting the predicted risk in descending order, so that the first 20% of patients represented the group with the lowest CVD risk. Among these quantiles, the predicted risk and observed risk were compared using a Hosmer-Lemeshow (HL) χ2 test. Moreover, the Brier score was calculated to assess the calibration of the model in terms of the mean squared difference between predicted risk and observed risk in the sample (range: 0–1). Patient and public involvement None. Results Out of 809 participants enrolled in the study, 60.3% were women. The mean age and T2DM duration were 55.9±11.5 and 7.64±11.5 years, respectively. Mean BMI and HBA1c were 30.71±6.37 kg/m2 and 7.4±1.6%, respectively. Albuminuria was present in 23.9% of participants, while 70.5% had HTN. Table 2 shows the sociodemographic and clinical characteristics of the study sample. The incidence of CVD events over the 5-year follow-up period was 4.6%, with MI being the most frequent type of CVD event (48.6%), followed by PAD (27%), non-fatal stroke (21.6%), unstable angina (8.1%) and stable angina (2.7%). No fatal cardiac or stroke events were reported during the follow-up period. Table 2Sociodemographic and clinical characteristics of adult Omani patients with type 2 diabetes mellitus attending primary healthcare institutions in Muscat Governorate (N=809)Characteristicn (%)/mean±SDFemale gender487 (60.2)Age in years55.9±11.5T2DM duration in years7.64±4.59Total cholesterol in mmol/L4.58±0.96HbA1c, %7.4±1.6BMI in kg/m230.71±6.37Albuminuria193 (23.9)HTN570 (70.5)CVD outcome37 (4.6)BMI, body mass index; CVD, cardiovascular disease; HbA1c, glycated haemoglobin; HTN, hypertension; T2DM, type 2 diabetes mellitus. Sociodemographic and clinical characteristics of adult Omani patients with type 2 diabetes mellitus attending primary healthcare institutions in Muscat Governorate (N=809) BMI, body mass index; CVD, cardiovascular disease; HbA1c, glycated haemoglobin; HTN, hypertension; T2DM, type 2 diabetes mellitus. In terms of the discrimination ability of the model, the AUC was 0.70 (95% CI 0.60 to 0.78). At a cut-off risk value of 11.8%, the model yielded a sensitivity and specificity of 67.6% and 66.5%, respectively. A cut-off CVD risk value of 10% yielded a sensitivity of 75.67% and specificity of 57.21%. Figure 1 shows the ROC curve of the model. Table 3 shows the diagnostic accuracy measures of the model at cut-off values of 10% and 11.8%. Table 3Performance* of a previously described cardiovascular risk model13 among adult Omani patients with type 2 diabetes mellitus attending primary healthcare institutions in Muscat Governorate (N=809)IndicatorCut-off value11.8%10.0%Sensitivity67.675.7Specificity66.557.2Positive likelihood ratio2.011.77Negative likelihood ratio0.490.43PPV8.87.84NPV97.798.0*Using a cut-off risk value of 11.8% and 10.0%.NPV, negative predictive value; PPV, positive predictive value. Performance* of a previously described cardiovascular risk model13 among adult Omani patients with type 2 diabetes mellitus attending primary healthcare institutions in Muscat Governorate (N=809) *Using a cut-off risk value of 11.8% and 10.0%. NPV, negative predictive value; PPV, positive predictive value. Receiver-operating characteristic curve (ROC) to determine the optimal cut-off risk value using a previously described cardiovascular risk model13 among adult Omani patients with type 2 diabetes mellitus attending primary healthcare institutions in Muscat Governorate (N=809). For the calibration assessment, the Brier score was reported to be 0.01, which may indicate good calibration of the model. However, the overall mean predicted risk was 11.8% compared with the overall observed risk of 4.6%, indicating risk overestimation by the model (p=0.15). Figure 2 shows a comparison of the predicted risks to observed risks in different quantiles. Bar chart comparing mean predicted and observed risk using a previously described cardiovascular risk model13 in quintiles of predicted risk among adult Omani patients with type 2 diabetes mellitus attending primary healthcare institutions in Muscat Governorate (N=809). HL, Hosmer-Lemeshow. Discussion This study aimed to externally evaluate the validity of a newly developed CVD risk assessment tool for Omanis with T2DM.13 Overall, the model showed good discrimination, but poor calibration. The discrimination ability of the model, which was tested using an ROC curve, showed an AUC of 0.70. The model was found to be applicable to both smokers and non-smokers, as there were minimal differences in the AUC for both groups. However, there was an overestimation of CVD risk when comparing mean predicted and observed risk. The ability of the model to predict CVD risk was assessed based on the area under the ROC curve. Compared with the internal validation study, which showed an AUC of 0.73 (95% CI 0.69 to 0.77), the current external validation study showed an AUC of 0.70 (95% CI 0.60 to 0.78).13 Previous research of CVD risk estimation models in T2DM populations in Australia (the ADVANCE model/the FDS model vs Framingham and UKPDS models), the UK (the DARTS model) and New Zealand (the DCS model vs Framingham and UKPDS models) have found similar AUC values of 0.68–0.71 satisfactory.16–19 In the current study, good sensitivity and specificity levels were reported for the model at a cut-off value of 11.8%. Similar to other international models of CVD risk, the negative predictive value of the model was very good, whereas the positive predictive value remained low.20–22 Such models are used primarily for screening rather than diagnostic purposes; therefore, a low positive predictive value is to be anticipated. Nonetheless, a high proportion of patients will be incorrectly classified as having a high risk of CVD. Patients with false-positive CVD risk results are often recommended by their physicians to modify their lifestyles and dietary habits and may also receive unnecessary and costly pharmacological preventative measures.23 24 Overestimation of risk within a risk assessment model or algorithm indicates poor calibration and often results in false-positive cases, thereby adversely affecting clinical decision-making. In this case, the overestimation of risk can be explained by the fact that this particular model was developed in a setting with a high disease incidence; as such, it may systematically overestimate risk when used in a setting where the incidence is lower.13 25 In addition, patient populations tend to change over time, for instance, due to changes in referral patterns or healthcare and treatment policies. As such, other methods should be used to further assess model calibration, for instance, average absolute error or an HL χ2 test, as was done in the present study (p=0.15).25 Overall, the performance of a risk prediction model depends on several factors, including the sample size, statistical methods used and the quality of the data collected.26 In the current study, all required CVD predictors were included; moreover, the study was reinforced using a cohort design, in addition to a sufficient sample size. Similarly, the data obtained may be considered of high quality as they were collected by professional, trained staff following a standardised data collection sheet. However, as it was designed for use as a simple screening equation, the model does not take into account other factors potentially affecting CVD risk, including the use of medications like angiotensin-converting enzyme inhibitors, other antihypertensive or lipid-lowering medications, statins or antihyperglycemic drugs.27 To some extent, use of such medications as preventive measures against CVD events might explain the overestimation of CVD risk seen in our study. In addition, the model does not consider lifestyle behaviours and other factors like socioeconomic status, which can have a considerable impact on cardiovascular health.28 29 Use of the model assessed in this study could potentially be extended to other countries in the Gulf Cooperation Council (GCC) due to the comparable lifestyle and ethnic make-up of populations in this region as well as the similarly high incidence and burden of CVD.30 However, future studies would be needed to validate the model in each of the nearby GCC countries due to the potential for population-level or healthcare system differences, which could render the model unsuitable for use in the studied patient group.27 31 Conclusion This study aimed to externally validate a newly developed cardiovascular risk model for Omanis with T2DM among a sample population in Muscat Governorate using multiple analysis methods. Although calibration assessment showed that the model magnified CVD risk among the sample groups, other tools showed that the model demonstrated good performance in terms of discrimination. However, additional calibration analysis and adjustment should be performed, after which the model has the potential to be applied in local settings. Supplementary Material Reviewer comments Author's manuscript