Assessment of Misspecification in CDMs using a Generalized Information Matrix Test
Reyhaneh Hosseinpourkhoshkbari and Richard M. Golden · 2024
Read Assessment of Misspecification in CDMs using a Generalized Information Matrix Test by Reyhaneh Hosseinpourkhoshkbari and Richard M. Golden (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2411-02769.txt.
Description
Abstract If the probability model is correctly specified, then we can estimate the covariance matrix of the asymptotic maximum likelihood estimate distribution using either the first or second derivatives of the likelihood function. Therefore, if the determinants of these two different covariance matrix estimation formulas differ this indicates model misspecification.
Catalog record
| CREATOR | Reyhaneh Hosseinpourkhoshkbari and Richard M. Golden |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 24 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2411-02769
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Assessment of Misspecification in CDMs using a Generalized Information Matrix Test Reyhaneh Hosseinpourkhoshkbari and Richard M. Golden Abstract If the probability model is correctly specified, then we can estimate the covariance matrix of the asymptotic maximum likelihood estimate distribution using either the first or second derivatives of the likelihood function. Therefore, if the determinants of these two different covariance matrix estimation formulas differ this indicates model misspecification. This misspecification detection strategy is the basis of the Determinant Information Matrix Test ( G I M T D e t GIMT_{Det} ). To investigate the performance of the G I M T D e t GIMT_{Det} , a Deterministic Input Noisy And gate (DINA) Cognitive Diagnostic Model (CDM) was fit to the Fraction-Subtraction dataset. Next, various misspecified versions of the original DINA CDM were fit to bootstrap data sets generated by sampling from the original fitted DINA CDM. The G I M T D e t GIMT_{Det} showed good discrimination performance for larger levels of misspecification. In addition, the G I M T D e t GIMT_{Det} did not detect model misspecification when model misspecification was not present and additionally did not detect model misspecification when the level of misspecification was very low. However, the G I M T D e t GIMT_{Det} discrimation performance was highly variable across different misspecification strategies when the misspecification level was moderately sized. The proposed new misspecification detection methodology is promising but additional empirical studies are required to further characterize its strengths and limitations. 0.1 Introduction Cognitive diagnostic models (CDMs) are a family of restricted latent class psychometric models designed to assess examinee skill mastery by incorporating prior knowledge of the relationship between latent skills and student responses ( Torre, 2009a ). In addition, a CDM outputs the attribute distribution or the probability that an examinee will have a particular set of skills given the examinee’s exam performance. Characteristics of latent skills in CDMs are defined through a Q-matrix that specifies which specific skills are relevant for answering a particular exam question. Furthermore, Q-matrix misspecification can affect parameter estimates and respondent classification accuracy ( Rupp & Templin, 2008a ). Therefore, careful validation of the Q-matrix is crucial to ensure that the model accurately represents the underlying relationships between the test items and the measured attributes. Methods have been developed to estimate and validate expert knowledge. However, despite best intentions, the possibility of CDM misspecification is always present. The effects of model misspecification in CDMs have been investigated by directly comparing observed frequencies and predicted probabilities ( Kunina-Habenicht et al., 2012a ), by comparing a correctly specified model to a nested model using Wald test methodology ( Torre, 2011a ), and comparing selected first and second order observed and expected moments ( Chen et al., 2018a ). These methodologies often face the challenge of test statistics with poor statistical power due to the difficulty of reliably estimating parameters in fully saturated (highly flexible models). For example, consider the problem of comparing the predicted probability of every pattern of responses to an exam with d items using a probability model with q free parameters. The degrees of freedom (and variance) for the Pearson Goodness-Of-Fit (GOF) chi-squared test statistic increases as an exponential function of the number of items d . The M2 statistic ( Maydeu-Olivares & Joe, 2014a ) provides an improvement in statistical power by only examining the first and second moments resulting in a chi-squared test statistic whose degrees of freedom increase as a quadratic function of the number of items d . Using a different approach, which detects model misspecification by comparing different covariance matrix estimators, White (1982a) introduced the Information Matrix Test (IMT) which is a chi-squared test with q(q-1)/2 degrees of freedom. This test is based on comparing the inverse covariance matrix estimators: \hat{{\bf A}}^{-1} , derived from the Hessian matrix, and \hat{{\bf B}}^{-1} , derived from the outer product gradient (OPG). These estimators are calculated from the second and first derivatives of the log-likelihood function, respectively. The White (1982a) IMT test statistic for misspecification detection has a chi-squared distribution whose degrees of freedom increase as a quadratic function of the number of parameters q in the model. Golden et al. (2013a) ; Golden et al. (2016a) proposed a generalization of the White (1982a) IMT framework called the Generalized Information Matrix Test (GIMT) framework. However, no systematic studies have been conducted to specifically investigate the performance of the GIMT in the context of model misspecification detection in CDMs. Presnell & Boos (2004a) developed and empirically evaluated a statistical test for comparing the ”in-sample” (training data) log-likelihood model fit to the ”out-of-sample” (test data) log-likelihood model fit. Presnell and Boos (2004) referred to their test as the ”in and out of sample” (IOS) test and showed the test statistic had a chi-squared distribution with 1 degree of freedom regardless of model or data complexity. The IOS test may be interpreted as a type of GIMT as described by Golden et al. (2013a) ; Golden et al. (2016a) which we call the Determinant GIMT ( GIMT_{Det} ) GIMT_{Det}=(1/q)\log\det(\hat{\bf A}_{n}^{-1}\hat{\bf B}_{n}) . More recently, Liu et al. (2019a) examined the performance of different methods in providing consistent standard errors (SEs) for item parameter estimates for situations where a CDM was misspecified or correctly specified. Liu et al. (2019a) showed a difference among OPG, Hessian, and Robust standard errors when the Q-matrix is misspecified. Although Liu et al. (2019a) had not intended to develop a method for misspecification detection, their empirical results nevertheless support a type of GIMT for misspecification detection as introduced by Golden et al. (2013a) ; Golden et al. (2016a) . In this paper, we describe another GIMT methodology that focuses on a single statistic for comparing covariance matrices rather than a comparison based upon their diagonal elements. First, we sketch the mathematical derivation of the asymptotic distribution of the GIMT_{Det} statistic for CDMs using the methods described by Golden et al. (2013a) ; Golden et al. (2016a) . Second, we empirically investigate the asymptotic behavior of the GIMT_{Det} for CDMs in a series of simulation studies. The simulation studies simulate data sets from a known DINA CDM fit to the Tatsuoka (1984a) Fraction-Subtraction data set. Next, the resulting bootstrapped simulated data sets are fit to the DINA CDM which generated the bootstrap data as well as different misspecified versions of the original DINA CDM. The discrimination performance of the GIMT_{Det} is then reported to provide an empirical evaluation of the ability of the GIMT_{Det} to detect model misspecification in a DINA CDM. 0.2 Mathematical Theory 0.2.1 Model Misspecification In the statistical machine learning framework, the Data Generating Process (DGP) generates observable data directly from an underlying unobservable probability distribution, p_{\text{DGP}} . A probability model, M , is a collection of probability mass functions. If p_{DGP}\in M , then M is correctly specified; otherwise, M is misspecified with respect to p_{\text{DGP}} . Less formally, a model capable of representing the DGP is called a ”correctly specified model.” 0.2.2 Cognitive Diagnostic Model Specification Data Set Consider a scenario where N