Log-linear Model for Dual System Estimation and Computational Considerations

Zhiyuan Lu · 2026

Read Log-linear Model for Dual System Estimation and Computational Considerations by Zhiyuan Lu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26607.txt.

Introduction

Zhiyuan Lu offers a faster route to log-linear estimates for dual-system population counts used in census work. Dual-system methods infer how many people each source missed, and log-linear fits via EM steps promise counts for every incomplete group. Yet the arithmetic blows up as populations split by state, ethnicity, and other factors, stalling joint analysis. Lu's alternative reaches the same maximum-likelihood estimates with far less computation, reopening multi-factor census estimation.

Catalog record

CREATORZhiyuan Lu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE63986
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26607

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Log-linear Model for Dual System Estimation and Computational Considerations Zhiyuan Lu Abstract Estimation of population sizes using multiple data sources has long been the practice of the Census Bureau. It is realistic to expect that even a census would not perfectly count every member of the population, and hence the use of a dual system estimator utilizing two independent population counts would provide more accurate figures. However, the use of multiple data sources cause issues when determining how much they agree. In particular, when dividing the population into distinct categories, the two sources can disagree on which category a particular person belongs to, or have missing category information on one or both sources for a recorded person. These issues have been described in the papers [ 2 ] and [ 1 ] . This existing work has provided the framework of a statistical model and recommendation of an EM algorithm, which allows the estimation of category sizes among two data sources. A downside of using the EM algorithm is the computational time scaling rapidly as the number of categories increase: the EM algorithm can obtain quick estimates in under 1 second if there are two data sources dividing the population into two categories each, but if there are two data sources each divided into 50 categories, then the computation can require hours. An alternative method that can obtain estimates much faster is thus greatly appreciated, as it will make the analysis of population counts divided into many groups much more feasible. This document will outline such a method. First the original statistical model and the EM algorithm will be described in detail, introducing notation that will be helpful in later sections. After this are sections describing the main mathematical results of this document, and conditions required for this result to hold. The most stringent condition required for the main result is that within the recorded data: • for contingency counts recorded on both data sources, all combinations of known categories have above 0 counts • for contingency counts recorded by one data source, all known categories of the recorded data source have above 0 counts. Under this and several other conditions, the primary new result written here will enable a new method, one that can be argued to obtain the same answer as using the EM algorithm while requiring much less computational time. This is also confirmed in simulations, where the EM algorithm and this new algorithm obtain very close estimates when applied to randomly generated data, while the newer algorithm runs in about 1/100 of the time. A proof of the main result will finally be presented at the end of the document. 1 Loglinear Model of Contingency Table for Two Data Sources The primary examples of contingency tables used in [ 2 ] and [ 1 ] involved two data sources, A and B , with data source A dividing its counts among the variable a and data source B dividing its counts by b . The survey data are summarized in the form of counts for x_{i,j,k,\ell} , where (i,j,k,\ell) are assorted values of (A,B,a,b) . Here, i and j can each take values of 0 or 1, corresponding to whether the count consists of data from register A or B , or missing from A or B . The category variables a and b can take a wider range of values, due to the potential of large number of categories and also due to counts from missing data entries. Here, the convention will be that a takes values from 1,\dots,n_{A} for counts without missing information from A , and -1,-2,\dots for counts with missing information from A . Similarly, b will take values from 1,\dots,n_{B} for counts with no missing information from B and -1,-2,\dots for counts of entries with missing information from B . B=1 (on MOH) B=0 (not on MOH) b=1 (non-Maori) b=2 (Maori) b=-1 (ethnicity unknown) b=-1 (ethnicity unknown) A=1 (on Census) a=1 (non-Maori) 3\,004\,335 31\,995 150\,840 38\,634 a=2 (Maori) 108\,189 435\,465 12\,405 4368 a=-1 (ethnicity unknown) 16\,512 2769 900 438 A=0 (not on Census) a=-1 (ethnicity unknown) 398\,838 146\,976 24\,636 - Table 1: Example of a contingency table of counts from two sources of data, a census and ministry of health (MOH) dataset taken from New Zealand, used in [ 2 ] . Here n_{A}=n_{B}=2 , there are 15 values of x_{i,j,k,\ell} for (i,j,k,\ell)\in Ind_{data} . The data set of contingency table counts is \big(x_{i,j,k,\ell}:(i,j,k,\ell)\in Ind_{data}\big) , for a collection of indices Ind_{data} . The aim is to take the available counts \big(x_{i,j,k,\ell}:(i,j,k,\ell)\in Ind_{data}\big) to estimate for the true counts \big(y_{i,j,k,\ell}:(i,j,k,\ell)\in Ind_{full}\big) , where Ind_{full}:=\{0,1\}^{2}\times\{1,\dots,n_{A}\}\times\{1,\dots,n_{B}\} are the indices of the complete contingency table the EM algorithm aims to estimate. B=1 (on MOH) B=0 (not on MOH) b=1 (non-Maori) b=2 (Maori) b=1 (non-Maori) b=2 (Maori) A=1 (on Census) a=1 (non-Maori) 3\,170\,294.8 33\,787.9 38\,616.0 411.6 a=2 (Maori) 111\,242.5 448\,084.8 877.6 3534.9 A=0 (not on Census) a=1 (non-Maori) 402\,709.4 10\,770.8 4905.2 131.2 a=2 (Maori) 14\,130.7 142\,839.1 111.5 1126.8 Table 2: Estimated values of y_{i,j,k,\ell} for the 16 values of (i,j,k,\ell)\in Ind_{full} , using the raw data of x_{i,j,k,\ell} ’s presented in Table 1 . These values were obtained from [ 2 ] . Usually, the index sets Ind_{full} and Ind_{data} are different sets with different indices as elements. In order to turn counts indexed by the former into counts indexed to the latter, a function is defined in order to link the collection of coordinates: CD(i,j,k,\ell):=\left\{\text{ indices in }Ind_{full}\text{ where the counts }x_{i,j,k,\ell}\text{ can be distributed to }\right\}. (1) In practical settings, if a coordinate \mathbf{u}\in Ind_{data} corresponds to a category without missing information, CD(\mathbf{u}) would just map to a singleton set (usually consisting of \mathbf{u} itself; if \mathbf{u} corresponds to a category with missing information, then CD(\mathbf{u}) maps to a set of all coordinates in Ind_{full} that counts in \mathbf{u} potentially can be distributed to. For example, in a data set where both registers A and B are split by 2 income levels (1-poor, 2-rich), then (1,1,1,-1) corresponds to someone recorded as rich in A but of unknown income in B . People counted in this category could be of either category in B , and hence a possible assignment is CD(\mathbf{u})=\{(1,1,1,1),(1,1,1,2)\} . (i,j,k,\ell) value CD(i,j,k,\ell) (i,j,k,\ell) value CD(i,j,k,\ell) (1,1,1,1) \{(1,1,1,1)\} (1,1,1,2) \{(1,1,1,2)\} (1,1,2,1) \{(1,1,2,1)\} (1,1,2,2) \{(1,1,2,2)\} (1,1,1,-1) \{(1,1,1,1),(1,1,1,2)\} (1,1,-1,1) \{(1,1,1,1),(1,1,2,1)\} (1,1,2,-1) \{(1,1,2,1),(1,1,2,1)\} (1,1,-1,2) \{(1,1,1,2),(1,1,2,2)\} (1,1,-1,-1) \{(1,1,1,1),(1,1,1,2),(1,1,2,1),(1,1,2,2)\} (1,0,1,-1) \{(1,0,1,1),(1,0,1,2)\} (1,0,2,-1) \{(1,0,2,1),(1,0,2,2)\} (1,0,-1,-1) \{(1,0,1,1),(1,0,1,2),(1,0,2,1),(1,0,2,2)\} (0,1,-1,1) \{(0,1,1,1),(0,1,2,1)\} (0,1,-1,2) \{(0,1,1,2),(0,1,2,2)\} (0,1,-1,-1) \{(0,1,1,1),(0,1,1,2),(0,1,2,1),(0,1,2,2)\} Table 3: Elements of Ind_{data} and which set they are mapped to using the CD function, in the context of the data and indices shown in Tables 1 and 2 . A loglinear model is assumed for the set of true counts \big(y_{i,j,k,\ell}:(i,j,k,\ell)\in Ind_{full}\big) : each y_{i,j,k,\ell} is generated from a Poisson distribution, with the mean determined by a set of variables corresponding to the maximal model described in [ 1 ] and [ 2 ] . This relation is expressed as \log\left(\mathbb{E}\left[y_{i,j,k,\ell}\right]\right)\sim\lambda_{0}+\lambda^{A}_{i}+\lambda^{B}_{j}+\lambda^{a}_{k}+\lambda^{b}_{\ell}+\lambda^{Ab}_{i,\ell}+\lambda^{Ba}_{j,k}+\lambda^{ab}_{k\ell}, (2) wtih the additional restriction that only the following variables are allowed to be nonzero: \displaystyle\lambda_{0},\lambda^{A}_{1},\lambda^{B}_{1}, \displaystyle\lambda^{a}_{k}\text{ for }k=2,\dots,n_{A}, \displaystyle\lambda^{b}_{\ell}\text{ for }\ell=2,\dots,n_{B}, \displaystyle\lambda^{Ab}_{1,\ell}\text{ for }\ell=2,\dots,n_{B}, \displaystyle\lambda^{Ba}_{1,k}\text{ for }k=2,\dots,n_{A}, \displaystyle\lambda^{ab}_{kl}\text{ for }k=2,\dots,n_{A},\quad\ell=2,\dots,n_{B}, (3) for a total of n_{A}n_{B}+n_{A}+n_{B} free variables. Another equivalent way

Read the complete text