Model-informed approach to estimate treatment effect in placebo-controlled clinical trials using an artificial intelligence-based propensity weighting methodology to account for non-specific responses to treatment Authors: Roberto Gomeni, F. Bressolle-Gomeni Published: 2024 ABSTRACT In randomized, placebo controlled clinical trials (RCT) in major depressive disorders (MDD), treatment response (TR) is estimated by the change from baseline at study-end (EOS) of the scores of clinical scales used for assessing disease severity. Treatment effect (TE) is estimated by the baseline-adjusted difference at EOS of TR between active treatments and placebo.The TE is function of treatment-specific and, non-specific (NSRT) effect (referred as placebo effect), and placebo response. The conventional statistical approaches used to estimate TE does not account for the potentially confounding effect of NSRT. This pragmatic approach is equivalent to assume that TE is independent of NSRT even if this assumption is not true, leading to potential risks of inflating false negative/positive results in presence of high proportion of subjects with high/low NSRT.The objective of this study was to develop a model informed framework to analyze the outcomes of RCTs using data driven models, non-linear-mixed effect approach, artificial intelligence, and propensity score weighted methodology (PSW) to control the confounding effect of treatment non-specific response on the estimated TE. The secondary objective was to explore the impact of relevant covariates (including the assessment of a dose-response relationship) on the outcomes of pooled data from two RCTs.The proposed PSW approach provides a critical tool for controlling the confounding effect of treatment non-specific response, to increase signal detection and to provide a reliable estimate of the ‘true’ treatment effect by controlling false negative results associated with excessively high treatment non-specific response.Supplementary InformationThe online version contains supplementary material available at 10.1007/s10928-024-09950-7. Introduction Non-specific responses to treatment (NSRT), commonly referred as placebo effect, represent a critical issue for the assessment of treatment effect (TE) of novel medications in CNS diseases [1]. The non-specific treatment effect is mainly due to expectations for improvement and the effect of receiving personal care from clinicians. The “specific effect” is usually associated with the clinical benefit only driven by the properties of the active drug used in the study. A high NSRT value has been shown to significantly compromise the signal detection of efficacy for new antidepressant medications, contributing to more failed randomized placebo-controlled clinical trials (RCTs) and delaying the delivery of new treatments. The increasing placebo response, preventing the assessment of the potential benefits of new medications, has been advocated as a reason to question the value of antidepressant medications, which may further stigmatize treatments for depression and dissuade patients from accessing mental health care [2]. The problem of identifying specific and non- specific responses is complex as these effects are latent and not directly measured. When a subject responds to a drug treatment, we cannot assess whether the subject responded to a specific or to a non- specific effect of a treatment. An additional complication is that a subject may respond to both specific and non- specific effects and the degree of each can vary from subject to subject. For this reason, there is an unmet need for developing novel methodologies to differentiate the specific from the non-specific responses in subjects treated with active drug. The treatment response (TR) in a RCT is usually estimated by the change from baseline at study-end (EOS) of the clinical scores used to assess disease severity (such as MADRS, or HAMD scales) and treatment effect (TE) is estimated by the baseline adjusted difference at EOS of TR between active treatments and placebo. The individual level of NSRT, together with the probability of experiencing a NSRT (prob-NSRT), has been shown to represent a major confounding factor in the analysis of the RCT outcomes [3]. Therefore, the individual prob-NSRT cannot be ignored in any inference for the assessment of TE and in the comparison of TEs across different RCTs even when randomization has been deployed [4]. The randomization procedure, applied to allocate subjects to treatments, is expected to mitigate potential selection bias. However, experimental data indicate that randomization is unable to insure a balanced baseline distribution of prob-NSRT across studies and arms [5]. Furthermore, the currently applied standard statistical methodologies for analyzing RCTs do not provide tools for accounting of potential baseline unbalance of relevant prognostic factors such as the level of NSRT. Hence, the estimated TE may represent either an under- or an over-estimated value as a function of the unknown baseline distribution of NSRT. Therefore, the uncontrolled NSRT factor compromises the possibility to estimate the ‘true’ TE and to establish a reliable correlation between dose/exposure and response. To address this issue, the use of a propensity weighted methodology (PSW) has been advocated as an effective tool to account for heterogeneity and unbalance in subject allocation to treatments [6–8] and for estimating a dose-response relationship [9]. The PSW methodology was recently applied for assessing TE conditional to the individual prob-NSRT value estimated using an artificial intelligence (AI) approach. The inverse of prob-NSRT was used as weight in the mixed-effect model to assess TE in RCT studies conducted in major depressive disorders (MDD) [5, 10, 11]. The idea behind the use of PSW is to control the influence of each subject enrolled in a RCT in the assessment of TE by weighting the individual response using the individual prob-NSRT values. The use of propensity scores to control for pretreatment imbalances on observed variables in non-randomized or observational studies examining the causal effects of treatments or interventions has become widespread over the past decade [12–15]. A sensitivity analysis was conducted to evaluate potential risk of inconsistent assessment of TE and study failure in new trials in presence of high/low level of non-specific responses to treatment by comparing the results of PSW and conventional analyses. This analysis indicated that the PSW methodology was associated with a reduced risk of inconsistent assessment of TE and study failure and that the PSW methodology provided better control of non-specific responses to treatment by providing estimates of TE independent from the distribution of the non-specific responses to treatment [5]. In MDD trials, the clinical response is usually evaluated using longitudinal clinical scores (e.g. HAMD). The trajectory of these curves shows a nonlinear decrement from a high initial values (e.g., ~ 23) to lower values within a time window of 6–8 weeks. It is generally accepted that HAMD values ranging from 7 to 10 indicate remission from MDD episodes. The nonlinear shape in the response and the presence of missing data represent a major problem in data analysis. To address these issues, different methods have been proposed. The conventional approach applied for the analysis of longitudinal clinical scores is based on the linear mixed-effects modeling approach for repeated measures (MMRM) [16]. The analysis is usually conducted using an unstructured covariance matrix, time as a classification variable, and baseline measurement as a covariate, baseline x time interaction, and treatment × time interaction with a significance level of α = 0.05 to establish the significance of the treatment effect. The non-linear mixed-effect modeling (NLMMRM) has been evaluated as an extension of the MMRM approach because this approach offers attractive advantages such as : (a) the possibility to characterize the clinical response using empirical or mechanistic models with parameters characterizing specific component of the longitudinal response, such as the time of the onset and/or duration of the response, (b) the higher flexibility in identifying the demographic, pathophysiological, environmental, genetic, PK, and drug related factors that contribute to explain the variability in the response, (c) the possibility to account for covariates effect to properly characterize the interindividual variability in the response, (d) the possibility to implement an integrated PK/PD model for establishing an exposure-response relationship, and (e) the possibility to implement clinical trial simulation to explore potential factors affecting study outcomes and the design of new RCTs. The comparison of the NLMMRM and the MMRM methodologies was conducted using data from 5 clinical trials including a total of 1837 MDD patients [17]. The individual longitudinal clinical scores (i.e. the HAMD-17 total scores) were described by a mixed Weibull/linear equation using the NLMMRM approach implemented in the NONMEM software or by analyzing the HAMD-17 changes from baseline scores using the MMRM approach implemented in the SAS PROC MIXED procedure. The comparison of the results of the analyses of the five clinical trials indicated that the two methodologies provided consistent results. A large number of empirical and semi-mechanistic models have been used to describe the longitudinal clinical scores in different CNS diseases [18–21]. Empirical models are data driven, do not account for underlying biological processes; however, they provide a mathematical framework for describing observed data, and are well suited for answering questions such as dose selection, assessment of clinical trial design and evaluation of the impact of covariates on study outcomes. The primary objective of this study was to develop a model informed drug development framework to analyze the outcomes of RCTs conducted in major depressive disorders using data driven models, non-linear-mixed effect approach, artificial intelligence model for estimating the NSRT, and the use of PSW methodology to account for the confounding effect of NSRT in the assessment of TE [22]. The secondary objective was to explore the impact of relevant covariates (including the assessment of a dose-response relationship) on the outcomes of pooled data from different RCTs. Methods Data The data of two antidepressant RCTs were used as cases-study. The first one, study 810, was a randomized, double-blind, parallel-group, placebo-controlled study evaluating efficacy and safety of paroxetine CR 12.5 mg/day (N = 156), paroxetine CR 25 mg/day (N = 154) versus placebo (N = 149) in patients with MDD and a mean age (± StdErr) of 39 (± 0.5) years. The primary efficacy endpoint was the change from baseline at week 8 in the 17-item Hamilton Rating Scale for Depression (HAMD-17) [23]. The second one, study 874, was a randomized, double-blind, parallel-group, placebo-controlled fixed-dose study evaluating the effect of of paroxetine CR 12.5 mg/day (N = 161), paroxetine CR 25 mg/day (N = 173) versus placebo (N = 178) in patients with MDD and a mean age (± StdErr) of 67 (± 0.3) years. The primary efficacy endpoint was the change from baseline at week 8 in the HAMD-17 scale [24]. Clinically relevant non-treatment specific response The clinical response was assessed using the longitudinal change of HAMD-17 total score [25]. The clinically relevant non-specific response was defined using the score of the Clinician Global Impression-Improvement (CGI-I) scale as this score represents a patient-independent assessment done by the clinician on the patient’s global disease severity prior to and after initiating the treatment. A CGI-I between minimally improved (= 3) and much improved (= 2) was considered as an indicator of a clinically relevant response. This CGI-I score was associated to a percent reduction from baseline at week 8 in the HAMD-17 total score of 41% using the equipercentile linking method [26–28]. A binary score (0 or 1) was then associated to each subject in the placebo arm as an indicator of the absence or presence of a clinically relevant signal of response at EOS when the change from baseline of HAMD-17 total score was greater than 41%. AI model development Six machine learning methodologies (gradient boosting machine, lasso regression, logistic regression, support vector machines, k-nearest neighbors, and random forests) were compared to the multilayer perceptrons artificial neural network (ANN) methodology for predicting the probability of individual non-specific treatment response. ANN achieved the highest overall accuracy among all methods tested. For this reason, the ANN method was retained as the reference methodology for estimating the individual prob-NSRT. The individual HAMD-17 items collected at two pre-randomization time points (i.e., at screening and baseline) were used to predict the prob-NSRT at EOS using an artificial neural network (ANN) methodology [29]. The implementation of an ANN model requires the definition of two hyperparameters that control the topology of the network: the number of hidden layers and the number of nodes in each hidden layer. A preliminary grid search analysis was conducted to identify these parameters using optimality criteria. These criteria were based on the best predictive performance of the ANN model defined by: The ROC AUC: Area under the Receiver Operating Characteristic curves with the 95% confidence intervals. The larger the ROC AUC the better the predictive performance of the model. The area of 1.0 is associated with the ideal predictor because it achieves both 100% sensitivity and 100% specificity. The area of 0.5 is associated with a poor predictor with 50% sensitivity and 50% specificity. A reliable and informative predictor should have an area statistically larger than 0.5.The Sensitivity or Recall: the ratio of true positives to total positives in the data.The Specificity: the ratio of true negatives to total negatives in the data.The Accuracy: the number of correct predictions as a fraction of total predictions.The Precision: the fraction of correct positive predictions. The ROC AUC: Area under the Receiver Operating Characteristic curves with the 95% confidence intervals. The larger the ROC AUC the better the predictive performance of the model. The area of 1.0 is associated with the ideal predictor because it achieves both 100% sensitivity and 100% specificity. The area of 0.5 is associated with a poor predictor with 50% sensitivity and 50% specificity. A reliable and informative predictor should have an area statistically larger than 0.5. The ROC AUC: Area under the Receiver Operating Characteristic curves with the 95% confidence intervals. The larger the ROC AUC the better the predictive performance of the model. The area of 1.0 is associated with the ideal predictor because it achieves both 100% sensitivity and 100% specificity. The area of 0.5 is associated with a poor predictor with 50% sensitivity and 50% specificity. A reliable and informative predictor should have an area statistically larger than 0.5. The Sensitivity or Recall: the ratio of true positives to total positives in the data. The Sensitivity or Recall: the ratio of true positives to total positives in the data. The Specificity: the ratio of true negatives to total negatives in the data. The Specificity: the ratio of true negatives to total negatives in the data. The Accuracy: the number of correct predictions as a fraction of total predictions. The Accuracy: the number of correct predictions as a fraction of total predictions. The Precision: the fraction of correct positive predictions. The Precision: the fraction of correct positive predictions. The ANN analysis was conducted using the ‘neuralnet’ library in R [30]. A formal procedure for the ANN model development and validation was applied. According to this procedure, the available data were split into three datasets: The training set, applied for the ANN model development (in our case this dataset included 75% of the data in the placebo arm randomly selected).The validation set used to assess the model performances in data not used for model development (in our case this dataset included the 25% data in the placebo arm not used for model development).The working dataset, with the full data set including all the subject data in the 3-arms together with the individual estimate of the propensity probability estimated using the ANN model applied to the pre-randomization data of each subject in each treatment arm. The training set, applied for the ANN model development (in our case this dataset included 75% of the data in the placebo arm randomly selected). The training set, applied for the ANN model development (in our case this dataset included 75% of the data in the placebo arm randomly selected). The validation set used to assess the model performances in data not used for model development (in our case this dataset included the 25% data in the placebo arm not used for model development). The validation set used to assess the model performances in data not used for model development (in our case this dataset included the 25% data in the placebo arm not used for model development). The working dataset, with the full data set including all the subject data in the 3-arms together with the individual estimate of the propensity probability estimated using the ANN model applied to the pre-randomization data of each subject in each treatment arm. The working dataset, with the full data set including all the subject data in the 3-arms together with the individual estimate of the propensity probability estimated using the ANN model applied to the pre-randomization data of each subject in each treatment arm. The ANN model, developed and validated using placebo data, was applied to the individual HAMD-17 item changes from screening to baseline of each subject in the three arms to estimate the individual prob-NSRT. Propensity score analysis Different propensity score methods have been proposed for removing the effects of confounders when estimating TE. Among these methods, we retained the inverse propensity score weighting (PSW), and the covariate adjustment using the propensity score [31]. The PSW methodology was finally retained as the main objective of the analysis was to assess the mean TE in a given population accounting for the individual estimate of the treatment-specific and non-specific responses. The inverse of the individual prob-NSRT was used as an estimate of the ‘true’ contribution of each subject enrolled in the trial for assessing TE. Differently from this approach, the covariate adjustment would have been more appropriate for explaining and controlling the inter-individual variability of the model parameters rather than for providing an adjusted mean TE estimate. Inverse probability of treatment weighting analysis The analysis was conducted in two stages. In stage 1, the data of the study 810 and 874 were independently analyzed for estimating the individual prob-NSRT using the ANN models. In stage 2, the data of the two studies were jointly analyzed using a PSW methodology to estimate the TR and the TE using a non-linear mixed effect modeling approach. The analysis was conducted using a 6-step approach: Step 1: The pre-randomization (i.e., screening and baseline) and end of study data (EOS) (i.e., visit on week 8) in subjects randomized to placebo were selected. Step 2: A predictive model was developed to estimate the probability to be placebo responder (prob-NSRT) after 8 weeks of treatment using the ANN model and pre-randomization data. Step 3: The model developed in step 2 was validated by comparing the model predicted probability to the observed placebo response. Step 4: The ANN model developed in step 2 was used to predict the individual prob-PE using the pre-randomization data of all subjects randomized in the study. Step 5: The individual prob-NSRTj value was estimated by applying the ANN model to the pre-randomization data of each subject enrolled in the trials. Step 6: The individual prob-NSRT value (prob-NSRTj) was included as individual weight in non-linear mixed effect (NLMMRM) model used for analyzing the longitudinal HAMD-17 total scores and for estimating the treatment effect (TE): higher is prob-NSRTj), lower will be the contribution of the jth subject to the TE estimation. Longitudinal model Two parallel independent analyses were conducted. The first one identified as ‘reference’ analysis was based on the conventional non-weighted non-linear regression analysis and the second one, identified as ‘propensity’, was based on the PSW methodology. The ability of the two analyses to assess the TE accounting or not for the presence of non-specific treatment response was assessed by comparing the results of the two analyses. A model describing the longitudinal time-course of the HAMD-17 total scores was used to jointly describe the pooled dataset of the 810 and 874 studies. This model data were described by the Weibull function: 1\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$P\:\left( t \right) = A{e^{ - {{(t/td)}^b}}} + {h_{rec}}t + eps$$\end{document} were P(t) is the HAMD-17 total clinical score at time t, A is the baseline HAMD-17 total score, td is the time corresponding to a 63.2% reduction of HAMD-17 from baseline, b is the rate of change in response, hrec is the recovery rate and eps is the residual error. This model was previously applied to describe the HAMD-17 longitudinal response in different RCTs conducted in MDD populations [17, 21]. As the primary endpoint of the two selected RCTs was the change from baseline of the total HAMD-17 score at week 8 (t = 8), the model was re-parameterized in order to include a parameter defining the HAMD-17 total score at week 8 (F8) and the change from baseline of the HAMD-17 total score (TR) at week 8: 2\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$F8 = A\:{e^{ - {{(8/td)}^b}}} + {h_{rec}}8$$\end{document} where 3\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:{h}_{rec}=\frac{F8-A\:{e}^{-(8/td{)}^{b}}}{8}$$\end{document} The change from baseline of the HAMD-17 total score (TR) at week 8 was estimated as: 4\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$TR = {\text{ }}F8 - A$$\end{document} The proposed model included three fixed-effect parameters: td, b, and F8. The value of A was fixed to the individual observed HAMD-17 total score at baseline. The inter-individual variability (IIV) model was assumed to be log-normally distributed. The relationship between a parameter P(td, b, and F8) and its variance was therefore expressed as: 5\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${P_j} = {P_{TV}}{e^{\eta p}}$$\end{document} Where Pj is the value of parameter for the jth individual, PTV is the typical value of P for the population, and \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\eta P$$\end{document} denotes the difference between Pj and PTV, normally distributed with a mean zero and variance\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\:\omega {P^2}$$\end{document}. The residual error model (epsj) for the ‘reference’ analysis was defined as epsj = ε, where ε is normally distributed with mean 0 and variance s2. The individual propensity (prob-NSRTj) for a non-treatment specific response was implemented in the longitudinal model as a subject specific weight in the residual error model (epsj). This model was defined as the ratio between components ε and Wj: 6\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$ep{s_j} = \varepsilon /Wj$$\end{document} where ε is normally distributed with mean 0 and variance s2 and Wj is the individual weighting factor defined as: 7\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$${W_j} = 1{\text{ }} - \left( {prob - NSR{T_j}} \right)$$\end{document} According to this model, a weight Wj was associated to each individual data: the subjects with lower Wj (reflecting higher uncertainty in the measurements for this individual) were considered as less informative and contributed less to the estimation of the model parameters and to the assessment of TE. The parameters were estimated using the NONMEM software (version 7.5.1) using the stochastic approximation expectation-maximization (SAEM) with the interaction computational algorithm. The − 2 loglikelihood (–2LL) value at the final model parameter estimates was calculated using the importance sampling approach (IMP). Model selection was based on likelihood-ratio test. An alternative nested model was considered statistically significant if it decreases − 2LL more than 3.84 (P < 0.05 with 1 degree of freedom). Covariate analysis The only prospectively identified covariates were dose and age. The impact of dose and age were evaluated on each of the three parameters of the longitudinal model: td, b, and F8. A dose = 0 was associated to the data in the placebo arm. The stepwise covariate model building procedure (SCM) was used to identify the relevant covariates [32] with a significance alpha level of 0.05 and of 0.01 for forward and backward selections, respectively. Longitudinal model evaluation The final ‘reference’ and ‘propensity’ models evaluation was accomplished by assessing the models for the goodness-of-fit plot, which included individual and population-predicted versus observed HAMD-17 scores, conditional weighted residual versus time and population-predicted HAMD-17 scores. The reliability and stability of the final longitudinal models were further assessed by a nonparametric bootstrap resampling method. Two thousand bootstrap datasets were generated by randomly sampling with replacements from the original dataset to form new datasets containing the same number of patients as the original dataset. The median parameter values and nonparametric 95% confidence intervals (CI) were derived from the NONMEM fit based on the 200 bootstraps dataset and compared with the final parameter estimates [13]. Furthermore, prediction-corrected visual predictive checks (pcVPCs) were performed to explore the predictive capability of the final longitudinal models [14]. The final model parameter estimates and the demographics information of subjects were used for simulating 200 new virtual clinical trials. The median and the 95% prediction interval simulated plasma concentration was plotted and compared with the observed data. Results A total number of 971 subjects were included in the pooled analysis dataset. A descriptive analysis on the age of the subjects indicated that the median age was 60 years with the first and the third quartiles of 39 and 66 years. Artificial intelligence model The initially conducted grid search analysis indicated that the optimal number of layers for the study 810 was 3 and the optimal number of nodes per layer was 12, 6, and 5. For study 874, the optimal number of layers was 3 and the optimal number of nodes per layer was 10, 13, and 3, respectively. All validation criteria indicated the good predictive performances of the models: Study 810: ROC AUC (95% CI) = 0.81 (0.64–0.97), Specificity = 0.88, Sensitivity = 0.75, Accuracy = 0.79, and Precision = 0.94; Study 874: ROC AUC (95% CI) = 0.88 (0.77-1.00), Specificity = 0.91, Sensitivity = 0.83, Accuracy = 0.88, and Precision = 0.88. The ROC AUC values were statistically greater than the non-informative threshold of 0.5 according to the 95% confidence interval. Therefore, the ANN models were considered appropriate for predicting the individual prob-NSRT using the pre-treatment data in each treatment arm of each RCT. The percentage of subjects with an estimated prob-NSRT in the intervals < 0.25, 0.25–0.50, 0.50–0.75, and > 0.75 is presented by study in Fig. 1. The distribution of prob-NSRT indicated a different pattern of response in the two trials. The large majority of the subjects in study 810 have a high prob-NSRT (> 0.75) leading to a potentially inflated Type II error (false negative) by underestimating the TE. On the contrary, the large majority of the subjects in study 874 have a low prob-NSRT (< 0.25) leading to a potentially inflated Type I error (false positive) by overestimating the TE. Fig. 1Distribution of the subjects with the probability of non-specific responses to treatment (prob-NSTE) in the four selected classes for the 810 ad 874 trials Distribution of the subjects with the probability of non-specific responses to treatment (prob-NSTE) in the four selected classes for the 810 ad 874 trials Longitudinal model An initial descriptive analysis was conducted to establish a base ’reference’ and ‘propensity’ models without accounting for the potential effect of covariates and without accounting for the individual prob-NSRT score. The scatter plots of the Empirical Bayesian estimates of the individual parameters versus the individual dose and age values were used to explore the potential impact of the covariates on the model parameters (Fig. S1 and S2 in the supplementary material). Given the distribution of the age in the analysis dataset, the age values were summarized in 3 classes: <40 years, between 40 and 70 years and > 70 years to better explore the relationship between age and model parameters. This analysis indicated a clear effect of age and dose on F8 and a borderline effect of age and dose on td. Therefore the effects of dose and age on the F8 and td parameters for the ‘reference’ and ‘propensity’ models were formally tested against the base model in a stepwise manner using the SCM procedure. The potential effect of dose on F8 was evaluated assuming a dose-related exponential reduction of F8: F8 = F8d*exp(-F8_dose*dose), F8_dose is a parameter defining the shape of the dose effect, and F8d is the value of F8 at dose = 0 (i.e., placebo). The potential effect of age on F8 was evaluated assuming a age-related linear reduction of F8: F8 = F8a + F8_age*age, were F8_age is a parameter defining the shape of the age effect, and F8a is value of F8 at age = 0. The potential effect of age and dose on the td parameter was evaluated assuming a age- and dose- related linear model. All tested intermediate models are listed in the supplementary Table S1 in the chronological order in which they were evaluated. The combined effect of dose and age on the F8 parameter was found to be statistically significant (p ≤ 0.05, df = 1) either for the ‘reference’ or the ‘propensity’ model. Therefore, the model implemented in the Run 6 was considered as the final model according to the pre-specified criterion on the likelihood-ratio-test. The population parameter estimates for the fixed-effect, the random-effect and the residual error model parameters in the final models are presented in Table 1 together with the precision of the parameters estimated using bootstrap. The good agreement between the parameter values estimated in the final NONMEM analysis and in the bootstrap procedure, together with the precision of the parameters, supports the adequacy of the ‘reference’ and ‘propensity’ models for describing the longitudinal HAMD-17 scores and to estimate the treatment response defined by the F8 parameter. The goodness-of-fit diagnostic plots for the ’reference’ and ‘propensity’ models are presented in the supplementary data (Fig. S3 and S4 for the ‘propensity’ and ‘reference’ models). Overall, there was no apparent bias in these diagnostic plots, suggesting that the models were adequate for describing the longitudinal HAMD-17 scores. The pcVPCs plots for the ‘propensity’ and ‘reference’ models are presented in Fig. 2 and the pcVPCs stratified by dose and study/age are presented in the supplementary material (Fig. S5 and S6 for the ‘propensity’ and ‘reference’ models). Fig. 2Prediction-corrected visual predictive checks for the ‘propensity’ and ‘reference’ models of the longitudinal HAMD-17 total score versus time computed using the final models. Points are observed HAMD-17 scores, solid red lines represent the median observed values, and dashed lines represent 5th percentiles and 95th percentiles of the observed values. The upper and lower shaded areas represent the 90% confidence interval spread of the 5th–95th percentile predicted values, and middle shaded areas represent the 90% confidence interval spread of predicted values Prediction-corrected visual predictive checks for the ‘propensity’ and ‘reference’ models of the longitudinal HAMD-17 total score versus time computed using the final models. Points are observed HAMD-17 scores, solid red lines represent the median observed values, and dashed lines represent 5th percentiles and 95th percentiles of the observed values. The upper and lower shaded areas represent the 90% confidence interval spread of the 5th–95th percentile predicted values, and middle shaded areas represent the 90% confidence interval spread of predicted values Simulations were conducted to explore the impact of dose and age on the change from baseline of the longitudinal HAMD-17 scores. In particular the mean expected change from baseline in the HAMD-17 score was estimated at the doses of 5, 12.5, 25, and 37.5 mg/day and at age of 40 and 70 years. Figure 3 shows the estimated longitudinal HAMD-17 changes from baseline scores for the ‘propensity’ model in the different scenarios in the left panel and the longitudinal change in the TE in the right panel. Fig. 3Propensity model. Left panel: estimated longitudinal HAMD-17 changes from baseline scores at the different doses and age levels. Right panel: longitudinal change in the TE at the different doses and age levels Propensity model. Left panel: estimated longitudinal HAMD-17 changes from baseline scores at the different doses and age levels. Right panel: longitudinal change in the TE at the different doses and age levels The total HAMD-17 change from baseline increases with the increase of the dose and the increase is larger in subjects with 40 years with respect to subjects with 70 years. The dose-response curve (Fig. 3 right panel) indicates a linear trend in the range of doses and ages evaluated with a greater slope for the elderly subjects at the higher doses explored. A forest plot with the summary results of the simulations conducted for the ‘reference’ and ‘propensity’ analyses is presented in Fig. 4. In this figure are reported the mean HAMD-17 TR values in the placebo and active arm on EOS (week 8) and the estimated TE with the 95% confidence interval. Fig. 4Forest plot of the outcomes of the ‘propensity’ and ‘reference’ models Forest plot of the outcomes of the ‘propensity’ and ‘reference’ models The comparison of the estimated TE in the ‘propensity’ and ‘reference’ models indicated that the ‘propensity’ model accounting for the non-treatment specific response provided an estimated TE significantly larger than the TE estimated with the ‘reference’ model. The mean percentage increase in the TE value estimated with ‘propensity’ with respect to the ‘reference’ method was 40% with a minimum and maximum value of 32% and 47%. Table 1Final parameter estimates in the population and in the bootstrap analysis. RSE represents the relative standard error and 95% CI the 95% confidence intervalPropensity ModelFinal modelBootstrap estimatesParameterEstimate(%RSE)95% CIMedian (2.5th -97.5th ) Fixed Effect tb (week) 5.20 (8.20)4.361–6.0394.95 (4.501–5.551) F8 10.90 (7.20)9.365–12.43510.47 (8.221–12.862) b 1.06 (3.40)0.99–1.131.087 (1.017–1.146) F8_age 0.0578 (24.0)0.031–0.0850.064 (0.030–0.098) F8_dose 1.33 (11.10)1.042–1.6181.461 (0.999–1.887) Random Effect tb 0.606 (18.50)0.386–0.8260.581 (0.474–0.695) F8 0.291 (8.80)0.241–0.3400.296 (0.258–0.332) b 0.426 (12.90)0.318–0.5340.413 (0.339–0.487) Res Error Add error 1.18 (4.60)1.075–1.2851.169 (1.085–1.265) Reference Model Final model Bootstrap estimates Parameter Estimate (%RSE) 95% CI Median (2.5th -97.5th ) Fixed Effect tb (week) 5.0 (4.50)4.555–5.4454.78 (4.43–5.17) F8 7.97 (7.50)6.794–9.1468.13 (6.75–10.08) b 1.07 (2.70)1.012–1.1281.09 (1.04–1.14) F8_age 0.0763 (15.10)0.054–0.0990.068 (0.036–0.096) F8_dose 1.04 (16.50)0.703–1.3770.915 (0.514–1.325) Random Effect tb 0.605 (12.20)0.460–0.750)0.564 (0.479–0.669) F8 0.359 (6.70)0.312–0.4060.362 (0.32–0.404) b 0.32 (10.80)0.252–0.3870.314 (0.264–0.387) Res Error Add error 2.32 (2.70)2.196–2.4442.313 (2.203–2.412) Table 1Final parameter estimates in the population and in the bootstrap analysis. RSE represents the relative standard error and 95% CI the 95% confidence intervalPropensity ModelFinal modelBootstrap estimatesParameterEstimate(%RSE)95% CIMedian (2.5th -97.5th ) Fixed Effect tb (week) 5.20 (8.20)4.361–6.0394.95 (4.501–5.551) F8 10.90 (7.20)9.365–12.43510.47 (8.221–12.862) b 1.06 (3.40)0.99–1.131.087 (1.017–1.146) F8_age 0.0578 (24.0)0.031–0.0850.064 (0.030–0.098) F8_dose 1.33 (11.10)1.042–1.6181.461 (0.999–1.887) Random Effect tb 0.606 (18.50)0.386–0.8260.581 (0.474–0.695) F8 0.291 (8.80)0.241–0.3400.296 (0.258–0.332) b 0.426 (12.90)0.318–0.5340.413 (0.339–0.487) Res Error Add error 1.18 (4.60)1.075–1.2851.169 (1.085–1.265) Reference Model Final model Bootstrap estimates Parameter Estimate (%RSE) 95% CI Median (2.5th -97.5th ) Fixed Effect tb (week) 5.0 (4.50)4.555–5.4454.78 (4.43–5.17) F8 7.97 (7.50)6.794–9.1468.13 (6.75–10.08) b 1.07 (2.70)1.012–1.1281.09 (1.04–1.14) F8_age 0.0763 (15.10)0.054–0.0990.068 (0.036–0.096) F8_dose 1.04 (16.50)0.703–1.3770.915 (0.514–1.325) Random Effect tb 0.605 (12.20)0.460–0.750)0.564 (0.479–0.669) F8 0.359 (6.70)0.312–0.4060.362 (0.32–0.404) b 0.32 (10.80)0.252–0.3870.314 (0.264–0.387) Res Error Add error 2.32 (2.70)2.196–2.4442.313 (2.203–2.412) Final parameter estimates in the population and in the bootstrap analysis. RSE represents the relative standard error and 95% CI the 95% confidence interval Discussion Development of pharmacotherapies for CNS disorders has become increasingly challenging due to the large failure rates of Phase II/III trials driven by the observed poor separation of clinical efficacy between subjects treated with active drugs and placebo. This finding was outlaid in a large meta-analysis conducted in RCTs with well-established marketed treatments used as positive control showing a rate of failure of ~ 50% [33]. One of the main reasons for the inefficiency of multicenter RCTs is the uncontrolled and excessively low/high level of treatment non-specific response. To overcome this issue, a propensity-weighted analysis was proposed. The basic assumption of this methodology is that subjects with high/low treatment non-specific response are less/more informative than the other subjects for estimating the ‘true’ treatment effect. Consequently, an individual “weight” of the clinical scores generated in each subjects was included in the analysis to estimate the TE accounting for the individual level of treatment non-specific response. Furthermore, the potentially imprecise estimation of TE generated without accounting for treatment non-specific response represents a clear issue for establishing a reliable dose-response relationship for antidepressant treatments [34]. For example, recent analysis based on either individual studies or meta-analyses was unable to demonstrate a clear dose-response relationship for antidepressant drugs selective serotonin reuptake inhibitors (SSRIs), leading to the conclusion that there is a flat dose-response curve for SSRIs [35]. The inability to properly characterize TE played a key role in the inability to establish a reliable dose/exposure-response relationship. As shown in the present analysis, the estimated TE value is function of the treatment non-specific response. As a consequence, the estimated TE in different RCTs can be different not only because the doses and/or the study design are different but also because the different RCTs have a different distribution of the treatment non-specific response. The primary objective of the present paper was to describe a new model-informed approach for analyzing RCTs conducted in MDD accounting for the non-specific treatment effect using an artificial intelligence driven PSW methodology. The secondary objective was to show how the PSW methodology provides a reliable approach for combining data from multiple RCTs, for exploring the impact of relevant covariates on the estimated TE, and for the assessment of a dose/exposure-response relationship. The ROC AUC analysis demonstrated that the ANN models provided a sufficiently precise estimate of the prob-NSRT in each RCT using data collected prior to randomization at screening and baseline. Two different ANN models were developed, one for each study, as the outcomes of the ANN model cannot be used for predicting prob-NSRT of the subjects enrolled in a different RCT. This because the individual prob-NSRT is driven by the individual expectations that is specific to each individual and specific to the interaction of each individual with the investigators at the different recruitment sites. The most relevant reason why the ANN model developed in one study (including the number of hidden layers and the number of nodes) cannot be used to predict the outcomes of a different study is associated with the definition of the expectations of improvement leading to an increased prob-NSRT. This is usually characterized by the adoption of an optimistic response criterion when deciding if the medication under evaluation is effective. This disposition is primarily generated by the interactions between the patients included in the trial and the investigator at the recruitment center. Confounding motivations, often implicit, may result in setting high expectations about the treatment outcome, with investigators desiring to heal the patient and patients wanting to please the investigators, leading to high improvement in the expectation. As investigators and patients varies study-by-study, the structure of the ANN model using the individual expectation will necessary be different study-by-study. The estimated individual prob-NSRT in the each RCTs provided a tool for data normalization with respect to the study-specific distribution of treatment non-specific response. This data normalization was instrumental for combining the data from different RCTs and for conducting a PSW pooled analysis that otherwise would have been potentially biased by the different distribution of treatment non-specific response. This because, the use of a PSW methodology has been advocated as an effective tool to account for heterogeneity and baseline unbalance in subject allocation to treatments [6, 8, 9]. Overall, our results indicated that the reassessment of TE using the PSW methodology improved efficacy signal detection of ~ 40% with respect to the reference approach by reducing the overall “noise” produced by subjects with a high propensity score and by reducing the impact of potential imbalances between treatment arms. The present paper presents a novel methodology for estimating TE using a Non-Linear Mixed-Effect Model for Repeated Measures and for assessing the impact of relevant covariates on the clinical response. This approach can be considered as the natural generalization of the likelihood-based mixed-effects model for repeated measures (MMRM) approach that is today recognized as the most efficient and reliable method for conducting the primary analysis of continuous endpoints in longitudinal clinical trials. The main limitation of this study is the restricted number of RCTs used to evaluate the performances of the PSW methodology. The results need to be replicated in other collections of clinical trials with different study designs, inclusion and exclusion criteria, placebo treatment durations, arm numbers, and dates of execution. Another apparent limitation is associate with the ability to provide an accurate and reliable estimate of the probability of treatment non-specific response using the ANN model. A reduced sample size in the placebo arm can certainly limit the accuracy of the ANN model to predict prob-NSRT. To overcome this potential risk, the analysis plan should specify the conditions for the acceptance of the ANN modeling outcomes: for example by indicating that the ANN modeling outcomes can only be accepted if the ROC AUC and the associate 95% confidence interval satisfie pre-defined criteria, such as ROC AUC > 0.75 and statistically greater than the noninformative threshold of 0.5. In conclusions, the proposed PSW approach provides an effective tool for controlling the confounding effect of treatment non-specific response, for increasing signal detection and for providing a reliable estimate of the ‘true’ treatment effect by controlling false negative results associated with excessively high treatment non-specific response. These features of the proposed approach can become critical to support strategic decisions on the progression in late phase clinical development or on the discontinuation of new medications based on limited information early collected in proof-of-concept studies. The proposed methodology is very simple to be implemented and the individual probability (prob-NSRT) can be easily computed and used in at least three scenarios: 1: in a post-hoc analysis conducted on data generated on any historical RCTs to re-assess the TE adjusted by the individual prob-NSRT. 2: prospectively use prob-NSRT in the analysis of new RCTs for controlling the effect of non-treatment specific effect. The prospective use of prob-NSRT would require to pre-specify in the statistical analysis plan the details of methodology to be used. 3: prospectively use prob-NSRT estimated using pre-randomization data as an enrichment criteria in future RCT trials to select a study population in which detection of a drug effect is more likely than it would be in an unselected population. The implementation of this scenario would require to dispose of a validated general ANN model linking pre-randomization measurements to prob-NSRT on large and pooled databases of RCTs. At the moment we are focused in the description of interest and of advantages of the methodology with the final objective to gain credibility and acceptance from a regulatory perspective. The findings of the analysis provide convergent evidences emphasizing the potential role of PSW to become the reference approach for analyzing placebo-controlled clinical trials. Electronic supplementary material Below is the link to the electronic supplementary material. Supplementary Material 1 Supplementary Material 1