External validation and refinement of the Psychosis Polyrisk Score to detect individuals at clinical high risk for psychosis Riccardo Stefanelli, Dominic Oliver, Stefano Damiani, Marika Orlandi, Umberto Provenzani, Elisa Bortolin, Margherita Rovida, Jean Addington, Luis Alameda, Celso Arango, Nicholas J Breitborde, Matthew Broome, Kristin S Cadenhead, Monica E Calkins, Ricardo E Carrion, Rolando I Castillo-Passi, Yu Hai Eric Chen, Jimmy Choi, Philippe Conus, Barbara A Cornblatt, Covadonga M Díaz-Caneja, Lauren M Ellman, Pablo A Gaspar, Carla Gerber, Louise Birkedal Glenthøj, Leslie E Horton, Christy LM Hui, Joseph Kambeitz, Lana Kambeitz-Ilankovic, Matcheri S Keshavan, Minah Kim, Sung-Wan Kim, Nikolaos Koutsouleris, Jun Soo Kwon, Kerstin Langbein, Daniel Mamah, Daniel H Mathalon, Vijay A Mittal, Meredete Nordentoft, Godfrey D Pearlson Abstract AbstractBackgroundThe Psychosis Polyrisk Score (PPS) was developed to characterise exposure to environmental and developmental risk factors for psychosis. In previous studies, the PPS showed promise particularly in discriminating between individuals at clinical high risk for psychosis (CHR-P) and community controls (CC).ObjectivesThis study aimed to (1) perform an external validation of the PPS model in a large independent sample and (2) refine the model using expanded predictor data.MethodsParticipants were recruited through the AMP SCZ program: n=1642 CHR-P, n=519 CC. We performed a formal external validation of the PPS original model. The model was then refined incorporating expanded predictor data (perceived loneliness, neighbourhood quality) and increasing the granularity of existing predictors (ethnicity, childhood trauma, parental severe mental illness). Next, we evaluated the refined model through repeated nested cross-validation and compared it to the original model retrained in the new dataset. Discrimination (Harrell’s C-index) and calibration (intercept and slope) were used to assess model performance.FindingsThe externally validated PPS original model showed good discrimination (C=0.777, 95% CI 0.756 to 0.797) between CHR-P and CC and good calibration (intercept=0; slope=1.003) after recalibration. The refined PPS model exhibited higher internally cross-validated discrimination than the retrained original model (C=0.876, 95% CI 0.860 to 0.891 vs C=0.796, 95% CI 0.775 to 0.816), as well as good calibration performance (intercept=−0.008; slope=1.031).ConclusionsThe PPS showed good generalisability to a large, independent sample, and performance improved after model refinement.Clinical implicationsThe PPS represents a potential scalable tool within stepped assessment frameworks for identifying individuals who warrant further CHR-P assessment.Trial registration numberNCT05905003. Introduction Early detection of individuals at clinical high risk for psychosis (CHR-P) is central to preventive psychiatry, given its potential to improve outcomes and reduce the societal burden of psychotic disorders. However, detection of CHR-P individuals is suboptimal. In clinical services, individuals with a first episode of psychosis (FEP) detected during their CHR-P stage are currently 5–14%,– despite 80% experiencing a detectable prodromal phase. This issue is mirrored following psychosis onset, with more than half of FEP patients being undetected by current early intervention frameworks. Given that recruitment strategies have a substantial impact on the level of risk within CHR-P samples,– it is crucial to develop tools that can facilitate large-scale screening without diluting risk. Psychosis risk can be conceptualised as the cumulative effect of multiple environmental and developmental factors that alter underlying neurobiology, whose aggregation may be more informative than any single exposure or symptom alone.– Within this framework, the Psychosis Polyrisk Score (PPS) was developed as a translational self-report instrument that integrates multiple environmental risk and protective factors based on the umbrella review by Radua et al into a single composite score, mirroring the logic of the polygenic risk score (PRS). The PPS captures key domains of environmental risk, including sociodemographic factors, migration status, childhood adversity, substance use, parental psychopathology and residential context (eg, urbanicity, pollution and ethnic density). As a scalable self-report tool, the PPS is intended to complement symptom-based prescreening instruments such as the 16-item Prodromal Questionnaire (PQ-16). In the European Network of Schizophrenia Networks for the Study of Gene-Environment Interactions (EU-GEI) study, the PPS explained over 30% of the variance between CHR-P individuals and healthy controls, compared with <2% for the PRS. Moreover, in formal prediction modelling analyses, the PPS showed excellent discrimination (Harrell’s C ≈ 0.91) between CHR-P and healthy controls in internal validation and displayed potential clinical net benefit of use in clinical samples. These results highlight the potential of the PPS as a scalable, pragmatic tool for CHR-P identification. However, there is a need for replication in independent, larger and more diverse populations to establish robustness and generalisability. Thus, the present study used the Accelerating Medicines Partnership Schizophrenia (AMP SCZ) program dataset to (1) externally validate the PPS original model developed from the EU-GEI study and (2) refine the model using expanded predictor data based on a recent update of Radua and colleagues’ umbrella review. The focus of both aims was discrimination between CHR-P individuals and a sample of community controls (CC). Methods Sample Participants were drawn from the AMP SCZ programme, which included the Psychosis-Risk Outcomes Network and the Prediction Scientific Global Consortium Research Networks, for a total of 38 sites across the globe. The project was approved by the governing institutional review board at each site. Participants and/or their legal representatives provided informed consent and/or assent as appropriate. For a broader overview of AMP SCZ procedure and structure, see Wannan et al. Analyses were conducted using AMP SCZ data Release V.3.0, which included de-identified baseline data from n=1642 individuals meeting CHR-P criteria and n=519 CC. For sample size calculation and inclusion/exclusion criteria, see . Measures The primary outcome was group status (CHR-P vs CC) at baseline (ie, a cross-sectional classification outcome, with no prospective time horizon). We evaluated two sets of predictors. - Predictor set 1 (PPS original model): predictors of the PPS original model developed by Oliver et al using the EU-GEI dataset. These predictors included: age, gender, ethnicity (white, black, other), handedness, first-generation and second-generation migrant status, North African immigrant background, paternal socioeconomic status, parental history of severe mental illness, recent adult stressful life events, daily tobacco use, weekly cannabis use, childhood trauma (total score only) and trait anhedonia. - Predictor set 2 (PPS refined model): an expanded predictor set aiming to improve the PPS original model, based on the update of Radua and colleagues’ review on psychosis risk/protective factors. Here, some variables were defined with higher granularity, including ethnicity (South Asian, East Asian, Southeast Asian, West-Central Asia/Middle-Eastern, Indigenous-Aboriginal, Mixed, Pacific-Islander), childhood trauma (sexual abuse, physical abuse, emotional abuse, physical neglect, emotional neglect) and parental history of severe mental illness (mother/father psychosis, mother/father other severe mental illness). Similarly, weekly cannabis use was replaced by daily cannabis use (to capture heavier use), and two new predictors were added: perceived loneliness and neighbourhood quality. Lastly, childhood trauma and anhedonia were considered as continuous predictors instead of using cutoffs as in predictor set 1. The full list of predictors, along with the instruments used to derive each of them, can be found in the (). Procedure The analytical approach proceeded in two steps: - External validation: for this step, the logistic regression coefficients estimated during model development using EU-GEI data were applied to predictor data (predictor set 1) from AMP SCZ. The aim was to assess the extent to which this original, already-developed model performed when applied to new, unseen data. - Model refinement: this step aimed to evaluate whether the inclusion of additional predictors (predictor set 2) enhanced model discrimination and calibration relative to the original model. However, due to the expected performance drop when externally validating a model, direct comparison with the external validation result above would provide an implicit advantage to the refined model. We therefore retrained the PPS original model within the AMP SCZ dataset (predictor set 1), estimating new coefficients. To improve conceptual clarity, we added a schematic overview of the analytical pipeline illustrating the distinction between the external validation of the PPS original model, the retrained original model and the refined model (). Statistical analyses Reporting was guided by the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis statement, and analyses were conducted in line with contemporary methodological guidance for prediction-model evaluation and updating. All analyses were performed in R (V.4.2). Analytic codes for model development and validation are available online (https://github.com/AMPSCZ). - External validation of the PPS original model (predictor set 1): the logistic regression coefficients from the EU-GEI-based PPS model were directly applied to the AMP SCZ dataset to compute individual-level predicted probabilities of CHR-P status. These were obtained by multiplying each coefficient calculated in the EU-GEI study by the raw score of the variables within the model. Variables with missingness >20% were excluded, whereas missing data <20% were imputed using random forest imputation (missForest R package, V.1.6.1). Primary outcomes were discrimination and calibration. Discrimination refers to the model’s ability to correctly distinguish between individuals with and without the outcome (ie, CHR-P vs CC), whereas calibration assesses how closely the predicted probabilities align with the observed outcomes. Discrimination was evaluated using Harrell’s C with 95% CIs obtained via 2000 bootstrap resamples. Calibration was assessed through the calibration intercept and slope, the Brier score and the Integrated Calibration Index (ICI). Threshold-dependent classification performance (ie, balanced accuracy, sensitivity, specificity, positive predictive value and negative predictive value) was measured across 100 evenly spaced probability cut-offs (0.01–1.00). The threshold yielding the highest balanced accuracy was chosen to report final performance metrics. To account for case-mix differences between the EU-GEI and AMP SCZ samples, the model was recalibrated by refitting a logistic regression with the PPS score as the sole covariate, thereby updating the intercept and slope to the new data distribution. Model performance was re-evaluated post-recalibration in the same way as above. - PPS model refinement and internal validation (predictor set 2): we fitted a refined version of the model using Least Absolute Shrinkage and Selection Operator (LASSO)-regularised logistic regression (glmnet R package, V.4.1.10) and repeated nested cross-validation (RN-CV). This technique framework combines two layers of cross-validation: an inner loop for hyperparameter tuning and model selection, and an outer loop for evaluating model performance on unseen data. By repeating this process multiple times with different random splits, RN-CV provides robust estimates of generalisation performance and minimises overfitting and optimism bias. In the present study, each RN-CV procedure employed five outer folds and five inner folds, repeated 10 times to ensure stability, mirroring the procedure of Oliver et al. Within each outer training fold, predictors with >20% missing values were excluded. The remaining missing data were imputed through random forest separately from any outcome data and independently in the training and test sets, ensuring no data leakage across folds. All predictors were standardised. Model performance was evaluated on the held-out outer test folds, yielding 50 out-of-fold estimates per metric (five outer folds×10 repeats). Out-of-fold predictions were then aggregated at the subject level by averaging predicted probabilities across repetitions. Overall model performance was computed on these aggregated predictions, and 95% CIs were estimated from empirical bootstrap distributions obtained through subject-level resampling (2000 iterations). Model performance was further evaluated across sites to ensure stability (leave-one-site-out cross validation). Sensitivity analyses were conducted to assess whether performance of the refined PPS model was dependent on predictors potentially affected by eligibility criteria, recruitment procedures or concerns regarding fairness and generalisability. First, we refitted the refined model after excluding age and parental psychosis, as CC were intended to be matched to CHR-P participants on age and were excluded if they had a full-blown first-degree relative with a psychotic spectrum disorder (as such, these predictors could potentially artificially inflate discrimination performance). Second, we refitted the model after excluding gender and ethnicity, given that these are sensitive sociodemographic predictors and, particularly for ethnicity, may partly reflect site composition or recruitment patterns in a large international multisite dataset. These predictors were retained in the main model because of their relevance in the psychosis-risk literature as well as potential importance for future external validations. Next, to enable appropriate comparison between the refined and the original PPS model, we retrained the original model using predictor set one in the new AMP SCZ dataset without a priori coefficients, using the same RN-CV procedure outlined above. We finally computed the logistic regression coefficients of the refined PPS model by fitting a final fivefold cross-validation on the full AMP SCZ raw dataset. These coefficients were used to compute new individual PPS scores, which were then correlated with subthreshold positive symptom severity (assessed using the PSYCHS) and socio-occupational functioning (assessed using the Social and Occupational Functioning Assessment Scale (SOFAS). Spearman’s ρ coefficient was used to account for non-normal distributions. Predictors with missingness >20% and coefficients shrunk to zero were excluded from the analysis. Standardised model coefficients with continuous variables were multiplied by two SD to increase comparability of variable importance across categorical and continuous predictors. The same procedure was used to compute the logistic regression coefficients of the retrained PPS original model. Lastly, we used decision curve analysis to evaluate the clinical utility of each model by estimating the net benefit of PPS-based classification across a range of threshold probabilities. These represent the minimum predicted probability of meeting CHR-P criteria at which a clinician would decide to refer an individual for further CHR-P assessment. For instance, a threshold of 0.20 implies that the potential benefit of assessing someone with a 20% risk outweighs the cost of assessing someone who ultimately does not meet CHR-P criteria. In this context, ‘treatment’ refers to offering a CHR-P clinical assessment (eg, Comprehensive Assessment of At-Risk Mental States (CAARMS)/Structured Interview for Psychosis-Risk Syndromes (SIPS)), rather than initiating any therapeutic intervention. We assessed clinical utility in clinical populations and the general population using threshold probabilities (clinical: 0–0.5; general: 0–0.3) and the meta-analytic prevalence of CHR-P in the relevant population (general: 1.72%; clinical: 19.2%). These two scenarios were used as lower-bound and upper-bound estimates, respectively, because the intended stepped-assessment use case is likely enriched relative to the general population but less enriched than specialist CHR-P clinical samples. The analysis was implemented using the dcurves package (V.0.5.1). Results Sample Most participants were assigned the female sex at birth, both within the CHR-P group (64.4%) and the CC group (61.5%). The mean age was 21.2±4.0 years for CHR-P individuals and 21.6±3.6 for CC. Most of CC (60.9%) and CHR-P individuals (54.8%) identified as females (gender), but within the CHR-P group a proportion of individuals were also non-binary (8.2%) or identified with other genders (1.8%). As for ethnicity, the majority was white (CC=52.3%, CHR-P=58.3%), with a fairly equal representation of the other ethnicities across the sample. All predictor statistics can be found in , whereas for a more in-depth overview of participants’ demographics and comorbidities see Addington et al. Lastly, provides an overview comparing the EU-GEI development dataset and the AMP SCZ validation dataset. External validation of the PPS original model The model demonstrated good discrimination in external validation (C-Index=0.777, 95% CI 0.756 to 0.797) (; ). However, calibration was suboptimal (intercept=−0.984; slope=0.947), indicating systematic overestimation of predicted probabilities (). After recalibration, calibration metrics improved (intercept=0; slope=1.003) (, ). Classification metrics across the 100 thresholds, for both the original and recalibrated model, can be found in the (). No variables needed to be excluded from this analysis due to missingness above 20% (). reports subgroup performance of the externally validated PPS model by gender and ethnicity. Refinement of the PPS original model The refined model demonstrated excellent performance (C-Index=0.876, 95% CI 0.860 to 0.891), with well-calibrated predictions (intercept=−0.008; slope=1.031) (, , ). Paternal age and childhood social withdrawal were excluded due to missingness >20% per outer fold (). Discrimination performance was 0.08 higher in the refined model compared with the retrained original model (C-Index=0.796, 95% CI 0.775 to 0.816) with evidence of improved calibration slope (retrained model’s slope=1.193). Full, per-fold classification and calibration indices of the refined and retrained model can be found in the Supplementary Material (, respectively). reports subgroup performance of the refined PPS model by gender and ethnicity. Model discrimination varied across sites, with greater variability observed in smaller sites, while larger sites showed more stable performance (mean C-Index=0.884±0.089) (). One site (Melbourne) contributed to a large proportion of participants (n=677). When removing age and parental psychosis, performance remained unchanged (C-Index=0.873, 95% CI 0.856 to 0.889) (). The predictors associated with the largest increase in being identified as CHR-P were loneliness and paternal psychosis (with betas >1.5). Negative coefficients were observed for South Asian (β=–0.62) and East Asian ethnicity (β=–0.32), though the absolute effect sizes were modest relative to positive predictors. The full list of coefficients and dropped variables are reported in , respectively. The coefficients of the PPS original model used for the external validation, as well as its retrained version, are displayed in . Finally, the PPS scores derived from the new logistic regression coefficients showed a weak association with the severity of subthreshold positive symptoms within the CHR-P group (ρ=0.238, p<0.001), and a negative association of similar magnitude with socio-occupational functioning (ρ=-.254, p<0.001) (). Decision curve analysis In clinical samples, the externally validated PPS model provided no incremental net benefit compared with a ‘treat all’ strategy (). In contrast, the refined and retrained models showed modest but consistent net benefits up to threshold probabilities of approximately 0.25 (). If a CHR-P assessment were considered necessary above a predicted probability of CHR-P status of 0.20 in clinical samples (ie, not detecting a CHR-P individual is four times as harmful as an unnecessary assessment), these two models demonstrated slightly higher clinical utility compared with the external validation model and the ‘treat all’ strategy, with the refined model performing better than the retrained model. In the general population scenario (), neither the externally validated nor the retrained PPS models showed any clinical utility, as their lines were indistinguishable from the ‘treat none’ and ‘treat all’ strategies. The refined model exhibited a modest advantage at extremely low thresholds but offered no net benefit at or above the general population prevalence threshold (0.0172), indicating limited practical utility in this context. Discussion This study presents the external validation and refinement of the PPS model, a tool designed to summarise environmental and developmental exposures associated with psychosis and to support classification of CHR-P status. Using data from the large, international AMP SCZ consortium, we evaluated the generalisability of the original PPS model to a large, independent sample and refined the model using expanded predictor information. Three major findings emerged: (1) the PPS original model retained good performance in the new AMP SCZ cohort (external validation); (2) the refined version of the model, based on the results of Aymerich et al’s meta-analysis, showed higher internally cross-validated performance than the retrained original model, which provides the fairest benchmark for estimating the incremental value of predictor refinement; (3) the PPS scores derived from the refined model showed a weak association with subthreshold positive symptoms and socio-occupational functioning within the CHR-P group. First, the external validation supported generalisability to AMP SCZ (C-Index=0.777), while also showing the need for recalibration, consistent with differences in case mix and predictor distributions between the two datasets (). The overall performance was lower than in the EU-GEI dataset, but this was expected and consistent with decreased model performance in external validation procedures due to context-dependent variations in predictor distributions and measurement properties. Second, refining the PPS model led to a significant improvement in its ability to detect CHR-P individuals relative to the retrained original model. The performance of the refined model was comparable to that of the original model in the EU-GEI dataset (C-Index ≈0.91). However, while performance deterioration in future external validations is expected, this is likely to be attenuated given the large sample size of the current study. Refining the PPS model highlights the added value of new predictors such as loneliness and increased granularity of other key predictors such as childhood trauma and parental mental illness. Loneliness, in particular, showed the largest positive coefficient for CHR-P classification, in line with prior systematic evidence. Similarly, emotional abuse and parental psychopathology were among the most influential predictors. Notably, the largest meta-analysis to date found that emotional abuse was the strongest contributor among trauma subtypes, associated with a more than threefold increase in the odds of psychosis. Having a parent affected by psychosis, on the other hand, was found linked to nearly sixfold higher odds of developing psychosis relative to a control offspring. For the interpretation of other coefficients, see . Third, and lastly, PPS scores were only weakly associated with greater subthreshold positive symptoms and poorer socio-occupational functioning. These findings suggest that, while accumulating environmental exposure may have measurable neurobiological consequences even before full psychosis onset, PPS-derived classification scores and subthreshold positive symptoms are related but distinct constructs. This supports the idea that PPS and symptom-based assessments (eg, CAARMS/SIPS) may complement each other within a stepped assessment framework, where the PPS represents a scalable, non-invasive measure of environmental and developmental exposures associated with CHR-P status, while symptom-centred interviews are reserved for individuals exceeding predefined classification thresholds. A similar framework is already being piloted in projects such as the E-DetectioN Tool for EaRly-stage at risk mental state in the community (ENTER) study. While ENTER is applied to the general population, other approaches may leverage the PPS by targeting other populations, including primary and secondary care. Alternatively, the PPS can be used to refine symptom-based prescreening measures such as the PQ-16. The use of decision curve analysis suggested that the PPS may have utility in enriched or help-seeking contexts, indicating that PPS-based classification could help streamline resource allocation by prioritising those most likely to meet CHR-P criteria. Conversely, none of the PPS models yielded clinical utility when assuming a general population context. While the CHR-P prevalence used (1.72%) might have been too low to enable meaningful clinical discrimination, these results emphasise the importance of prescreening within the general population. In this sense, a three-stage pathway based on symptom-centred prescreening tools such as the PQ-16, followed by a systematic investigation of environmental risk factors using the PPS, and a final semistructured interview (CAARMS/SIPS/PSYCHS) may represent an effective strategy. Whether the PPS could also be deployed for stratification within already-identified CHR-P samples remains instead an open question. Taken together, these findings do not support use of the PPS as a standalone tool for general-population screening. Rather, they suggest that any potential clinical utility is more plausible in enriched or help-seeking contexts, where CHR-P prevalence is expected to be higher than in the general population but lower than in highly enriched CHR-P clinical samples. Despite its promising findings, the present study has some limitations to be acknowledged. First, the cross-sectional design does not permit establishing causal or temporal relationships between predictors and CHR-P status. Some predictors, including loneliness, anhedonia and social withdrawal, may also reflect features of the emerging prodromal state rather than antecedent environmental risk factors, raising the possibility of reverse causation. Second, the case-control design precludes inference about the absolute probability (or real-world incidence) of CHR-P status, as the proportion of cases and controls is deliberately fixed. Consequently, predicted probabilities derived from the model may be biased or inflated when applied to general population settings, where CHR-P status prevalence is substantially lower. The case-control design may also be susceptible to spectrum bias in that sampled individuals may differ markedly from those in true clinical settings, exaggerating the contrast between groups. In this regard, it is important to highlight that the control group of this study was a community sample with some individuals reporting past or current comorbidities including depression or passive suicidal ideation. As such, although the case-control design may exaggerate the distinction between groups in some respects, the inclusion of less stringently defined controls may have attenuated observed differences between CHR-P individuals and CC. The present findings, including sample-dependent metrics such as positive and negative predictive value (PPV/NPV), should be therefore interpreted as evidence of generalisability to the AMP SCZ validation context rather than definitive evidence of generalisability across all clinical, cultural or geographical settings. Third, several predictors relied on retrospective, self-reported measures and are therefore prone to recall bias. Fourth, not all theoretically relevant exposures could be included due to data availability (eg, urbanicity, ethnic density and pollution). Fifth, although methodologically conservative, the exclusion of predictors with missingness >20% (ie, paternal age and childhood social withdrawal) did not allow assessment of the potential incremental value of these variables, limiting the generalisability of the final predictor set to datasets in which paternal age or premorbid social withdrawal are measured. Future external validations should test whether these variables contribute additional predictive information when available with lower missingness, particularly given prior umbrella-review evidence reporting childhood social withdrawal and paternal age among risk factors for psychotic disorders. Sixth, the effects of certain environmental exposures (particularly ethnicity) may not be uniform across countries or sociocultural contexts. Globally developed models such as the refined PPS could benefit from local adaptation to maintain optimal performance. Lastly, the refined PPS model was internally validated using RN-CV but has not yet been externally validated in an independent cohort. External validation of the refined model in other datasets is therefore required before stronger claims regarding its broader generalisability can be made. In conclusion, this study supports the generalisability of the original PPS model to a large, independent sample. Refined through the addition of new and more granular predictors, the PPS showed improved discrimination and calibration performances, although its clinical utility appears most plausible in enriched or help-seeking contexts rather than as a standalone general-population screening tool. Future work should externally validate the PPS refined model and assess its longitudinal utility in predicting transition to psychosis, along with its feasibility in stepped/sequential assessment frameworks for CHR-P detection and stratification across more diverse settings. In such pathways, use of the PPS would require trained clinical or research users to collect the relevant predictor data, apply the scoring algorithm and interpret the resulting score as part of a broader CHR-P assessment process.