Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples.
Austin PC. · 2009
Read Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples. by Austin PC. (2009) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/europmc-pmc3472075.txt.
Introduction
Austin set out clear ways to check whether matched groups in a medical study are truly alike, in Statistics in Medicine in 2009. Researchers often match treated and untreated patients by a propensity score, an estimate of each person's chance of getting the treatment. Any claim about the treatment holds only if the matched groups share similar starting traits.
The paper walks through checks based on standardized differences, variance ratios, fuller summaries, and plots that compare whole distributions. Austin also derives how large a difference can arise by chance when matching has worked. He warns that comparing the scores themselves between groups tells nothing useful. The guidance helps readers judge whether a matched study was built soundly.
Catalog record
| CREATOR | Austin PC. |
|---|---|
| YEAR | 2009 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 69 KB |
| LICENSE | CC BY 4.0 |
| RIGHTS | CC BY |
| SOURCE | Europe PMC |
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Balance diagnostics for comparing the distribution of baseline covariates between treatment groups in propensity-score matched samples By Peter C Austin Published in: Statistics in Medicine DOI: 10.1002/sim.3697 ABSTRACT Abstract The propensity score is a subject's probability of treatment, conditional on observed baseline covariates. Conditional on the true propensity score, treated and untreated subjects have similar distributions of observed baseline covariates. Propensity-score matching is a popular method of using the propensity score in the medical literature. Using this approach, matched sets of treated and untreated subjects with similar values of the propensity score are formed. Inferences about treatment effect made using propensity-score matching are valid only if, in the matched sample, treated and untreated subjects have similar distributions of measured baseline covariates. In this paper we discuss the following methods for assessing whether the propensity score model has been correctly specified: comparing means and prevalences of baseline characteristics using standardized differences; ratios comparing the variance of continuous covariates between treated and untreated subjects; comparison of higher order moments and interactions; five-number summaries; and graphical methods such as quantile–quantile plots, side-by-side boxplots, and non-parametric density plots for comparing the distribution of baseline covariates between treatment groups. We describe methods to determine the sampling distribution of the standardized difference when the true standardized difference is equal to zero, thereby allowing one to determine the range of standardized differences that are plausible with the propensity score model having been correctly specified. We highlight the limitations of some previously used methods for assessing the adequacy of the specification of the propensity-score model. In particular, methods based on comparing the distribution of the estimated propensity score between treated and untreated subjects are uninformative. Copyright © 2009 John Wiley & Sons, Ltd. Keywords: balance, goodness-of-fit, observational study, propensity score, matching, propensity-score matching, standardized difference, bias 1. INTRODUCTION Researchers are increasingly using observational studies to estimate the effects of treatments and exposures on health outcomes. In randomized controlled trials (RCTS), randomization ensures that, on average, treated subjects will not differ systematically from untreated subjects in both measured and unmeasured baseline characteristics. Non-randomized studies of the effect of treatment on outcomes can be subject to treatment-selection bias in which treated subjects differ systematically from untreated subjects. Propensity-score methods are being used with increasing frequency to estimate treatment effects using observational data. The propensity score is defined as the probability of treatment assignment conditional on measured baseline covariates [1, 2]. Rosenbaum and Rubin demonstrated a key property of the propensity score: conditional on the true propensity score, treatment status is independent of measured baseline covariates [1]. In other words, treated and untreated subjects with the same propensity score will have similar distributions of observed baseline covariates. Four methods of using the propensity score have been described in the statistical literature: covariate adjustment using the propensity score, stratification on the propensity score, propensity-score matching, and inverse probability of treatment weighting using the propensity score [3]. In the medical literature, the first three are used almost exclusively [4–6]. Propensity-score matching, which is the focus of the current paper, is used frequently in the medical literature [7–9]. It entails forming matched sets of treated and untreated subjects with similar values of the propensity score. The most common implementation of propensity-score matching is 1:1 matching in which matched pairs of treated and untreated subjects are formed. Rubin argues that an advantage to the use of propensity-score methods is that it allows observational studies to be designed similar to randomized experiments: the design of the study is separated from the analysis of the effect of exposure on the outcome [10]. Rubin states that ‘diagnostics for the successful design of observational studies proposed on estimated propensity scores … is a critically important activity in most observational studies’ [11]. In randomized or designed experiments, the propensity score is frequently known and fixed by the study design. However, in observational studies, the true propensity score is not known and must be estimated using the study data. The propensity score is a balancing score: conditional on the true propensity score, treated and untreated subjects have the same distribution of observed baseline covariates. Ho et al. have defined the propensity-score tautology as follows: ‘we know we have a consistent estimate of the propensity score when matching on the propensity score balances the raw covariates’ [12]. In other words, Ho et al. are suggesting that one has adequately specified the propensity-score model when, after matching on the estimated propensity score, the distribution of measured baseline covariates is similar between treated and untreated subjects. Therefore, the appropriateness of the specification of the propensity score is assessed by examining the degree to which matching on the estimated propensity score has resulted in a matched sample in which the distribution of measured baseline covariates is similar between treated and untreated subjects. We use the term balance diagnostics to describe methods for assessing whether the distribution of baseline covariates is similar between different exposure groups. In the context of propensity-score matching, balance diagnostics enable applied researchers to assess whether the propensity-score model has been adequately specified. The test of adequate specification is whether matching on the propensity score has removed observed systematic differences between treated and untreated subjects. Descriptions of balance diagnostics in the context of propensity-score matching are scattered across the statistical, medical, and social sciences literature. The objective of the current study is three-fold. First, to summarize in one paper previously described methods for assessing balance in measured baseline covariates between treated and untreated subjects in propensity-score matched samples. Second, to provide a framework for assessing the adequacy of the specification of the propensity score model. Third, to highlight limitations of some previously suggested methods for assessing the adequacy of the specification of the propensity-score model. The paper is structured as follows. In Section 2, we describe the data that were used for illustrating the different methods for assessing balance in baseline characteristics. In Section 3, we describe methods for comparing the mean of continuous variables or the prevalence of dichotomous baseline covariates between treated and untreated subjects in the propensity-score matched sample. In Section 4, we use Monte Carlo simulations to illustrate that obtaining balance in means alone can be insufficient. We then describe additional methods for comparing the distribution of baseline covariates between treated and untreated subjects in the matched sample. In Section 5, we highlight limitations of some previously proposed methods for assessing balance. Finally, in Section 6, we summarize our findings. 2. DATA SOURCES AND PROPENSITY-SCORE MATCHING METHODS 2.1. Data sources We used data on 9104 patients who were discharged alive following hospitalization with a diagnosis of acute myocardial infarction (AMI or heart attack) from 102 hospitals in Ontario, Canada, between April 1, 1999 and March 31, 2001. These data are similar to those reported on elsewhere [13–15], and were collected as part of the Enhanced Feedback for Effective Cardiac Treatment (EFFECT) Study, an initiative focused on improving the quality of care for cardiovascular disease patients in Ontario [16]. Data on patient demographics, presenting signs and symptoms, classic cardiac risk factors, comorbid conditions and vascular history, vital signs on admission, and results of laboratory tests, were abstracted directly from patients’ medical records. The exposure of interest was whether the