Nonparametric Assessment of the Calibration of Individualized Treatment Effects Authors: Mohsen Sadatsafavi, Jeroen Hoogland, Thomas P. A. Debray, John Petkau Published: 2026 ABSTRACT ABSTRACTAn important aspect of the performance of algorithms that predict individualized treatment effects (ITEs) is moderate calibration, that is, the average treatment effect among individuals with predicted treatment effect of z being equal to z. The assessment of moderate calibration is a challenging task on two fronts: counterfactual responses are unobserved, and quantifying the conditional response function for models that generate continuous predicted values requires regularization. Perhaps because of these challenges, there is currently no inferential method for the null hypothesis that an ITE model is moderately calibrated in a population. In this work, we propose nonparametric methods for the assessment of moderate calibration of ITE models for binary outcomes using data from a randomized trial. These methods simultaneously resolve both challenges, resulting in novel graphical, numerical, and inferential methods for the assessment of moderate calibration. The key idea is to formulate a stochastic process for the cumulative prediction errors that obeys a functional central limit theorem, enabling the use of the properties of Brownian motion for asymptotic inference. We propose two approaches to construct this process from a sample: a conditional approach that relies on predicted risks (often an auxiliary output of ITE models), and a marginal approach based on replacing the cumulative conditional expected value and variance terms with their marginal counterparts. Numerical simulations confirm the desirable properties of both approaches and their ability to detect miscalibration of different forms. We use a case study to provide suggestions on graphical presentation and the interpretation of results. Moderate calibration of predicted ITEs can be assessed without requiring regularization techniques or making assumptions about the functional form of treatment response. The accompanying cumulcalib R package implements this method ( https://cran.r‐project.org/package=cumulcalib). Introduction Due to their direct relevance to clinical decision‐making, prediction algorithms that produce quantitative estimates of individualized treatment effect (ITE) have been the subject of much recent development [1, 2, 3]. A common paradigm in medical practice is to offer treatment to patients at high risk of untoward clinical outcomes, where the risk is quantified via the application of prognostic markers or risk prediction models [4]. Yet, evidence that these patients actually benefit more than those with low outcome risk is often lacking. Indeed, this risk‐based approach toward treatment recommendation is often based on strong and untested assumptions such as a monotonic association between treatment effect and outcome risk. By directly producing an estimate of treatment effect given an individual's salient characteristics, ITE models enable more principled decision‐making. An example of such a model is the GRACE 3.0 ITE predictor for early invasive management of non‐ST‐elevation acute coronary syndrome [5], developed using data from the VERDICT trial. Another example is the ITE model for the effect of oxygen‐saturation targets on mortality in patient under intensive care, based on data from the PILOT trial [6]. Before being implemented for patient care in a population, the performance of an ITE model should be assessed using an independent sample from that population—a practice commonly referred to as (external) validation [7, 8]. A critical aspect of ITE model performance in a validation study is calibration: the ability of the model to generate predictions that are close to their true counterparts [9]. Poorly calibrated ITE models may yield individualized predictions that are exaggerated or do not adequately reflect actual benefit. Consequently, they can result in misguided clinical decisions by care providers and may mislead patients in making treatment choices that are not aligned with their preferences. In the context of risk prediction models, calibration is defined at different levels [10, 11, 12]. The most basic form is calibration in the large (a.k.a. mean calibration), which refers to the average predicted and average observed risks being equal. Of more interest is assessing whether average observed risk among all individuals with a given predicted risk is equal to that predicted risk. This form of calibration is interchangeably referred to as “weak” (e.g., by Pepe et al. [10] and Blanche et al. [12]) or “moderate” (e.g., by van Calster et al. [11]) calibration. Henceforth we use moderate calibration for consistency with our directly‐related previous work [13]. This terminology is to contrast it with “strong” calibration, which refers to the average observed and predicted risks being equal within every subgroup defined by a combination of predictor values [10, 11, 12]. However, it is argued that strong calibration is neither achievable nor is it strictly desirable [11]. This makes mean calibration (often the very first step in calibration assessment) and moderate calibration of particular relevance. In practice, mean calibration is assessed by comparing the average observed and predicted risks, while moderate calibration is often assessed visually using a flexible calibration plot, which depicts the conditional mean of observed risk as a function of predicted risk. When predicted risks are continuous with no or few ties, estimating this conditional mean requires some form of smoothing or grouping. Scalar metrics have also been proposed to summarize it [14, 15, 16]. Because these visualizations and summary metrics rest on an estimated conditional‐mean function, their computation depends on tuning parameters (e.g., the number of bins, or the LOESS bandwidth and polynomial degree), and the choice of such parameters can have a noticeable impact on the assessment. Principles of calibration hierarchy can largely be extended to ITE models [9]. Mean calibration can be examined by comparing the observed average treatment effect (ATE) and the average of predicted effects. For weak/moderate calibration, van Klaveren et al. [17] quantified the difference between predicted and observed absolute treatment effects in quartiles of predicted effects. Maas et al. [18] proposed applying local smoothing functions to match‐based estimates of conditional treatment effect, producing graphical presentations and numerical metrics for the assessment of moderate calibration. Hoogland et al. [8] proposed fitting logit models for outcome risk separately within control and treatment groups, from which the calibration function for ITEs can be derived. However, these approaches require grouping of data into bins, applying local smoothing methods, or using parametric regression modeling. Calibration can also be approached as a formal inference problem. Inference on mean calibration for a risk prediction model entails testing whether the average predicted risk equals the average observed risk. Traditionally, inference on moderate calibration for risk prediction models has been based on the Hosmer–Lemeshow test, albeit this test is criticized for being sensitive to the inevitably arbitrary number of groups (often set to 10) [19]. For ITE models, mean calibration can be tested using standard methods (e.g., testing for the equality of the average observed and predicted treatment effect). However, to the best of our knowledge, there are currently no inferential methods available for testing whether an ITE model is moderately calibrated. This is an important methodological gap, as achieving moderate calibration is an important criterion for many clinical applications to proceed with the implementation of the model or assessment of impact in further studies. Miscalibrated models might result in loss of clinical benefit, as a treatment decision based on the predicted ITE might be different from the decision had the true ITE been known. Recently, nonparametric approaches have been proposed for assessing moderate calibration of risk prediction models based on the behavior of cumulative prediction errors that do not require smoothing or grouping of observations [13, 20, 21]. The core idea is that if the model is calibrated, the cumulative sum of prediction errors should stay near zero, whereas for miscalibrated models, the cumulative errors will tend to drift away from zero along their path. These approaches yield a new class of visualizations, summary metrics, and inferential tools for moderate calibration, without requiring regularization. In particular, they enable formal inference on the null hypothesis that “this model is moderately calibrated in this population,” based on asymptotic tests that rely on known properties of Brownian motion. It would be appealing to extend this line of work to the assessment of calibration of ITE models. This extension is challenging, as, unlike risks, treatment benefits are not typically observable at the individual level. The purpose of this work is to propose non‐parametric, tuning‐parameter‐free approaches for the assessment of moderate calibration of ITE models. Because of their causal underpinnings, ITE models are often trained using data from randomized controlled trials (RCTs). As such, we focus on validation studies based on RCT data, and, as a first step, we consider binary outcomes. The rest of this manuscript is structured as follows: after outlining the context, we briefly review nonparametric calibration assessment for predicted risks. We then extend this idea to ITE models by creating a stochastic process that converges asymptotically to standard Brownian motion, which is the basis for two approaches for constructing this process in the sample. We conduct simulation studies examining the performance of tests based on both approaches. These developments are exemplified in a case study where we showcase the assessment of moderate calibration on the cumulative domain. We conclude by discussing how these methods can be added to the toolbox for evaluating ITEs and suggest avenues for future research, including their use in non‐randomized settings. Methods Notation and Context We focus on binary outcomes (e.g., in‐hospital mortality due to a severe COVID‐19 infection), and assume the outcome is an untoward event, and, as such, treatment effect is in terms of absolute risk reduction. For treatments that increase the probability of a desirable outcome, all subsequent developments are applicable after relabeling the treatment and control interventions. Let Y(0) and Y(1) be the potential outcomes under control and treatment, respectively, both taking a value in {0,1}. Let Y be the observed response, taking a value in {0,1}. The treatment assignment indicator is a (0: control, 1: treatment). An ITE model is a deterministic function that for a given patient maps their covariate pattern to a predicted treatment effect δ, taking a value in (−1,1), which is claimed to be a good approximation for EY(0)−Y(1) among all individuals with the same covariate pattern. The ITE model can be a regression‐based or machine‐learning algorithm (as the functional form of the mapping algorithm is not of direct relevance). Many ITE models are based on risk prediction models that separately make predictions for outcome risk under each treatment scenario, either from separate models (e.g., a T learner) or a model with treatment as a predictor (a S learner) [22]. These models therefore yield counterfactual predictions: the expected outcome if treated and the expected outcome if receiving the control treatment among all individuals with the same covariate patterns. The predicted ITE is the difference between these predicted values. For such models, by π we refer to the “baseline” risks, which are claimed to be a good approximation for EY(0). However, some modeling approaches can generate predicted ITEs without modeling the underlying risks [23, 24]. For our validation task, we assume we have access to an iid sample of size n from a parallel‐arm RCT comparing the treatment and control of interest. For the ith individual, we have access to the following items: ai, representing treatment assignment, the binary response variable Yi, and the predicted ITE δi. For ITE models that also generate predicted risks, we will also have access to the predicted baseline risk πi (which takes a value in (0,1)). We consider δs and πs (if the ITE model also produces them) fixed at their calculated values. While not strictly necessary, for ease of derivations, we assume no ties in predicted benefits (i.e., at least one of the predictors is a continuous variable). Our goal is to construct a stochastic process {S}, that is, a process with an implied time and an implied location component, based on the cumulative sum of prediction errors that asymptotically converges to Brownian motion, W(t), on the [0,1] time interval, if the ITE model is moderately calibrated. This enables us to extend the previously developed nonparametric approaches for calibration assessment of risk prediction models to ITE models. Nonparametric Approaches for Moderate Calibration of Predicted Risks In this section, we briefly review nonparametric methods for the assessment of moderate calibration of algorithms that predict outcome risks given an individual's predictor pattern. These methods form the basis of our subsequent developments for ITE models. To avoid introducing new notation, in this section we assume none of the individuals has received the treatment. As such, the baseline predicted risks π=π1,π2,…,πn remain pertinent predicted risks for the observed responses. For such predicted risks, the null hypothesis that the model is moderately calibrated is defined as ∀z:E(Y|π=z)=z. If a model generates continuous predicted risks, estimating the conditional mean E(Y|π) requires regularization (binning or smoothing). The core idea of nonparametric assessment is that one can instead examine moderate calibration on the cumulative sum domain. We start by indexing observations after ordering them from the small to large values of the predicted risks πs. We then construct the scaled cumulative sums of the prediction errors. For the kth observation, this is (1) Ck=1n∑i=1kYi−πi,fork=1,2,…,n. It follows immediately from the definition of moderate calibration that ∀k:ECk=0 under the null hypothesis. This expectation can be estimated from the data, without any need for regularization. If the model is not moderately calibrated, the Cks tend to deviate substantially from zero along their path. Tygert proposed plotting Ck as a function of k/n to inspect such deviation [20]. The degree of this deviation can be summarized into scalar metrics. One such metric, proposed by Arrieta‐Ibarra et al. [21], is the scaled maximum absolute value of cumulative prediction errors: (2) C*=maxk|Ck|. This metric summarizes the degree of miscalibration, and can be interpreted as the maximum absolute cumulative prediction error, as an alternative to scalar summary metrics of miscalibration such as Emax and the integrated calibration index (ICI) [14, 16]—one whose computation does not involve any tuning parameters (e.g., LOESS bandwidth) and thus results in an objective, reproducible summary of miscalibration. Importantly, Arrieta‐Ibarra et al. argued that under the null hypothesis that the risk prediction model is calibrated, the distribution of a properly scaled version of C* converges, as n→∞, to the distribution of the maximum absolute deviation of Brownian motion: nC*/∑i=1nπi1−πi→dsupt∈[0,1]|W(t)|. This is the test statistic of their proposed asymptotic test for moderate calibration (henceforth referred to as the Brownian Motion [BM] test). Neither Tygert nor Arrieta‐Ibarra et al. explicitly discussed constructing a stochastic process—a process with a time and a location component—for prediction errors. The implied process in Ck, with time increments of 1/n and location changes of (Y−π)/n, is heteroscedastic, as the variances π(1−π) are variable but the time increments are all equal. As such, while certain summaries of this process converge to their counterparts for W(t) under H0, this implied process does not achieve point‐wise convergence to W(t). Sadatsafavi and Petkau [13] proposed turning these cumulative sums into a stochastic process that converges at all points to W(t), thus opening the door for taking full advantage of the properties of the latter. Let sk2=∑i=1kπi1−πi be the cumulative sum of variances up to the kth observation. They defined the following standardized process: (3) {S}=tk=sk2/sn2(time values)Sk=nCksn(location values) (with t0=0 and S0=0). The time increments are scaled to sum to 1, and the variances of the location changes are scaled to be equal to the time increments. This process is, under H0, a martingale with independent and strictly bounded increments, with sn2→∞ as n→∞. Thus, Brown's martingale CLT establishes that the continuous process based on linear interpolation of successive (tk,Sk) points converges weakly to W(t) on the [0,1] time interval [25]. As an immediate application of this convergence, they proposed a two‐part “bridge” test. This test is motivated by the following observations. First, under H0, Sn→dW(1)∼Normal(0,1). Thus, Sn, the test statistic for the first part (based on the terminal position of the random walk), is a z‐score for mean calibration, a necessary condition for moderate calibration. Second, the path of a “bridged” Brownian motion, which is defined as the difference between W(t) and the line that connects its endpoints (also known as the Brownian bridge [26]) is independent of its terminal value: ∀t∈[0,1):W(t)−tW(1)⊥W(1). As miscalibrated models tend to generate large deviations from this path, examining the maximum absolute deviation of the bridged random walk provides an opportunity, independently of Sn, for assessing moderate calibration. The test statistic for the second part is thus S*=maxk |Sk−tkSn| , with S*→dsupt∈[0,1]|W(t)−tW(1)| under H0. The distribution of the maximum absolute deviation of the Brownian bridge is the Kolmogorov distribution, the asymptotic null distribution of the two‐sided one‐sample Kolmogorov–Smirnov test statistic [27, 28]. The p values of each part are combined via Fisher's method to generate a unified p‐value for moderate calibration. Simulations confirmed the intuition that, being based on two independent statistics, the bridge test generally has higher power than the BM test. The authors also proposed graphical visualization of the homoscedastic {S} process as an alternative to the presentation of the heteroscedastic scaled cumulative calibration errors (C). Figure 1 provides examples of the visualization of the {S} process and typical miscalibration patterns. While this exemplary process is built around a risk prediction model, the interpretation of the process will be identical for ITE models. Example of the {S} process for a calibrated model* (A), a model that underestimates the actual outcome (B), and an overfitted model that makes extreme predictions (C). *The construction and interpretation of the S process are the same for risk‐prediction and ITE models: The qualitative reading of the trajectory as random fluctuation (A), systematic drift (B), or an inverse‐U departure (C), carries over directly. The true model is logit(P(Y=1))=−1+X with X∼Normal(0,1). The three prediction models are as follows: (A) Same as the true model; (B) logit(P(Y=1))=−1.2+X; (C) logit(P(Y=1))=−1.2+1.2X. Results are from a random dataset of 1000 observations. Time and Location axes refer to their corresponding elements in Equation (3). Note the random variation of cumulative errors in Panel A (typical of a calibrated model), the systematic increase in Panel B (indicating a systematic underestimation of risks), and the inverse U‐shape pattern in Panel C (indicating an over‐dispersed model that makes overly extreme predictions—underestimating risks when they are low and overestimating them when they are high). Nonparametric Approaches Toward Calibration of ITE Models Our main purpose is to generalize the above‐mentioned developments to the assessment of the calibration of ITE models. Before proceeding to main developments, we note that for ITE models that predict risks under control and treatment, predicted ITEs are the difference between these predicted risks. If predicted risks under both treatment conditions are to be communicated to the patient and be used in decision‐making, it is natural to demand that they both be moderately calibrated [8]. For this purpose, one can define a compound null hypothesis as H0A:∀z:EY(0)|π=z=z and H0B:∀z:EY(1)|π=z=z−δ. In the context of a randomized trial, and under classical assumptions for the potential outcomes framework (consistency, exchangeability, positivity, and no interference [29]), these hypotheses can also be written as H0A:∀z:E(Y|a=0,π=z)=z and H0B:∀z:E(Y|a=1,π=z)=z−δ. Because the subgroups of treated and control individuals are independent of each other, this compound null hypothesis can be tested via separate nonparametric assessment of the calibration of predicted risks within each treatment group. A unified p‐value can be generated via Fisher's method. A visualization of this approach would involve plotting the {S} processes within each treatment group. However, this approach fails to focus the attention on the component of interest here which is ITE calibration. Indeed, the ITEs can remain calibrated despite miscalibrated predicted risks. The visualization of cumulative predicted risk errors also does not provide an opportunity for a direct assessment of the cumulative prediction errors for ITEs. Further, this approach is not applicable to ITE models that do not output predicted risks. As such, for the rest of this work we study stochastic processes that focus on the calibration of predicted ITEs. In this context, the null hypothesis that an ITE model is moderately calibrated can be formulated as follows: (4) H0:∀z:E(Y|a=0,δ=z)−E(Y|a=1,δ=z)=z. A Stochastic Process for Cumulative ITEs Our goal is to create a standardized {S} process for cumulative ITEs with weak convergence to W(t), enabling the use of the above‐mentioned visualizations, summary indices, and inference for the assessment of ITE models. To transform the assessment domain to cumulative prediction errors, we start by indexing the observations after ordering the sample from small to large values of predicted ITEs, such that ∀i predicted ITE, resulting in positive errors). The curve then declines, indicating that the model over‐estimates the benefit for larger predicted ITEs (actual ITE < predicted ITE, resulting in negative errors). As noted earlier, this is a signature of an over‐fitted model that makes exaggerated predictions: it over‐predicts benefit for patients with high predicted ITE and under‐predicts it for those with low predicted ITE. Correspondingly, the summary indices are large, indicating a substantial degree of miscalibration, and the bridge p values are small, indicating strong evidence for miscalibration. In such instances, the location of C* provides useful auxiliary information: taking the conditional approach in the large validation sample as an example, the maximum absolute cumulative deviation occurs at a predicted ITE of −0.0034. Predicted ITEs around this value are approximately calibrated, whereas those below it are downwardly biased (benefit under‐estimated) and those above it are upwardly biased (benefit over‐estimated). Visualization of the {S} process for the small‐development‐sample model, an example of a clearly miscalibrated ITE model. Rows and columns are as in Figure 7. Axes and plot elements are as described in Figure 7. Here the standardized path rises and then falls (an inverted U): Prediction errors accumulate positively for small predicted ITEs (benefit under‐estimated) and negatively for large ones (benefit over‐estimated)—the signature of an over‐fitted model that exaggerates the spread of predicted ITEs. The large C* and small bridge‐test p values (Table 3) confirm strong evidence of miscalibration. Model coefficients (β0–β11) are, respectively: −2.2635, 0.4721, 0.0529, −0.5550, 0.4923, 0.4515, 0.6840, −0.0622, 0.0210, −2.3426, −1.4945, 0.0435. TABLE 3Small‐development‐sample model validation results.Validation sampleApproach Cn (mean calibration error) C*(maximum absolute cumulative error) p‐value (bridge test)LargeConditional0.02230.0328 < 0.0001LargeMarginal0.02120.0325 < 0.0001SmallConditional0.03640.0488 < 0.0001SmallMarginal0.03370.0464 < 0.0001 Note: Columns are as in Table 2: Cn is the (signed) mean calibration error and C* the (non‐negative) maximum absolute cumulative calibration error (both on the risk‐reduction scale), and the p‐value is that of the two‐part bridge test. Small‐development‐sample model validation results. Note: Columns are as in Table 2: Cn is the (signed) mean calibration error and C* the (non‐negative) maximum absolute cumulative calibration error (both on the risk‐reduction scale), and the p‐value is that of the two‐part bridge test. Discussion In this work we proposed novel nonparametric methods for the assessment of moderate calibration of predicted ITEs for binary outcomes. These methods are applicable when a pre‐specified ITE model is being validated in a new sample. We have demonstrated that moderate calibration can be assessed without resorting to regularization if calibration is examined on the cumulative sum domain. We proposed an estimator for cumulative ITE errors and showed that a properly scaled version of this estimator will result in a stochastic process that, under the null hypothesis that the ITE model is moderately calibrated, converges point‐wise to the standard Brownian motion. We showed that the conditional moments of this estimator depend on the true outcome risks. For ITE models that predict risk, such predictions can be used under the condition that they are calibrated, giving rise to our proposed “conditional” approach. However, the investigator might not be willing to assume those risks are calibrated; indeed, not all ITE models generate predicted risks. For these instances, we proposed an approximate “marginal” approach that solely relies on predicted ITEs. These developments enabled us to extend the previously proposed visualization, summarizing indices, and tests for moderate calibration of predicted risk to the predicted ITE domain. In a case study we laid out our suggestions for the visualization of cumulative prediction errors and the interpretation of proposed tests. The proposed tests are implemented in the cumulcalib R package (https://cran.r‐project.org/package=cumulcalib) with exemplary code provided in Section S2. For inference, our proof‐of‐concept simulations confirmed the desired properties of the resulting tests. Simulations under the null hypothesis, where ITE predictions were moderately calibrated, showed that these asymptotic tests generally maintain their type I error rate, even with sample sizes of 500, which we consider modest for an ITE model validation study. In simulations where the null hypothesis was violated, the power of these tests increased with larger sample sizes and with more severe miscalibrations, albeit the power could be low if the true calibration function was close to the identity line. The two‐part (bridge) test had generally more power than the one‐part test (at times significantly), an unsurprising finding given that this test offers two independent opportunities for detecting miscalibration. Because it rests on between‐arm differences, assessing ITE calibration is more demanding, in terms of sample size, than assessing risk calibration. This is related to the notion that detecting treatment‐by‐covariate interaction requires significantly larger sample sizes [32]. In our simulations the tests were well‐powered at n=10,000 and detected pronounced miscalibration at n=2,500, but had limited power at n=500. In modest samples, therefore, high p values, especially in the presence of a large C*, should not be interpreted as evidence in favor of calibration; rather, that the sample does not offer strong evidence against the claim that the model is calibrated. We generally advise against significance testing at an arbitrary level for model validation. Rather, p values are better interpreted as continuous measures of the compatibility between the data and the hypothesis of moderate calibration, complementing, not replacing, the reported magnitude of miscalibration [33]. How can these developments be used in practice? Some models generate ITEs from underlying predicted risks under alternative treatment decisions. Such predicted risks might be used for decision making in tandem with predicted ITEs. Correspondingly, one might demand that the model generate calibrated predictions across all its outputs. Requiring an ITE model to be calibrated on both predicted risk and ITE domains is equal to requiring it to be calibrated on predicted risks within each treatment group. One option is therefore to assess calibration separately for risks within each treatment group. However, this does not put the focus on calibration of ITEs, which might be of independent interest. Indeed, miscalibration of risks within each group might cancel out and result in calibrated ITEs; reciprocally, otherwise subtle miscalibrations of risks can augment and result in more pronounced miscalibration of ITEs. These patterns might not be obvious when predicted risks are assessed independently. Therefore, even for ITE models that also predict risk we propose an assessment of the calibration of ITEs, especially the plot of the {S} process. If the predicted risks are deemed calibrated (or the model is modified to correct for miscalibration), one can then use the conditional approach, for which a functional CLT could be proven. However, the marginal approach also seems to have desirable properties while not requiring the assumption of calibrated predicted risks. The latter remains the only option for models that generate predicted ITEs without modeling risks. We propose visualization of the standardized cumulative calibration plot, as well as reporting Cn, that is, mean calibration error, and C*, that is, maximum absolute cumulative prediction error. If inference is desired, we suggest the use of the two‐part “bridge” test over the one‐part BM test (all these suggestions are exemplified in our case study). Investigators often assess mean calibration (by comparing the predicted and observed ATEs) before moderate calibration. As mean calibration is a necessary condition for moderate calibration, such sequential inference alters the power of assessing moderate calibration. The bridge test provides an opportunity for joint inference that preserves the desired overall type I error rate while reducing type II error rate because of its higher power. Further, in many instances, if mean calibration assessment is not satisfactory, one might first decide to address the miscalibrated ATEs before embarking on moderate calibration assessment. For example, for an ITE model that predicts risks under control and intervention strategies using a logit link function, one might fix a biased ATE prediction by adjusting the intercept of the logit functions separately within each treatment arm. This approach results in average predicted risks exactly matching the average observed risks within treatment groups, in turn leading to an exact match between the predicted and observed ATEs. In these instances, the one‐part BM test is not a valid inference method, as the terminal value of the {S} process is guaranteed to be zero. In such instances, one can apply the bridge component of the bridge test in isolation to assess moderate calibration conditional on mean calibration having been achieved. There are several areas for future research. As a first step, we proposed these approaches for binary outcomes and when the sample is from a randomized experiment. An important extension, with wide applicability, would be for other outcome types, in particular survival outcomes. For such outcomes, the challenge is that the observed response is not only a function of predicted risk, but also the distribution of censoring events and the shape of the hazard function. As for extending this approach to non‐randomized samples, the challenge is that a, δ, and Y can be dependent in such settings, violating the martingale properties underpinning the proposed stochastic processes. A potential path forward is the application of inverse‐probability‐of‐treatment weighting schemes that can result in pseudo‐populations where treatment assignment can be considered random. However, the application of a functional CLT when observations are weighted would require additional technical developments. Further, other known properties of the Brownian motion can be used to propose alternative tests. In addition to the test based on maximum absolute deviation, Arrieta‐Ibarra et al. [21] proposed a test based on the maximum range of the Brownian motion (max(W)−min(W)). In the same vein, a test based on the maximum range of the Brownian bridge will result in a new two‐part test. Another potential test would be based on the square of the area under the Brownian motion and Brownian bridge. The resulting tests might have more power as they capture the behavior of the {S} process across its entire path. Lastly, our simulation studies were aimed at investigating the expected properties of the proposed tests. The power of these tests in detecting specific types of miscalibrations should be investigated in dedicated simulation studies. This should include a deeper investigation of the consequences of predicted risks not being calibrated for the conditional approach, and investigating the marginal approach in more varied settings given its heuristic underpinnings. Relatedly, while our framework conditions on the model's predicted ITEs and is thus agnostic to how they were generated (and the proposed marginal approach does not require arm‐specific predicted risks to be available), our simulations produced predicted ITEs as differences of arm‐specific predicted risks. Assessing predictions from estimators that target the treatment effect directly (e.g., causal forests or R‐learners) is a further useful direction for future work. A further extension of this framework is to study calibration of predictions across the domain of another variable. Conventionally, as in this work, we assess calibration as the closeness of predicted and actual quantities across the strata of predicted quantities (ITEs in our case). However, this closeness can also be investigated across the strata of another variable. For example, it is often reasonable to investigate whether a risk prediction or an ITE model makes calibrated predictions among individuals of different age. This involves assessing the closeness of predicted and observed risks or ITEs within strata defined by age. The core idea of nonparametric calibration assessment is to transport the conditional mean EY(0)−Y(1)−δ|δ=z into the cumulative domain EY(0)−Y(1)−δ|δ