Learning from Neighbors with PHIBP: Predicting Infectious Disease Dynamics in Data-Sparse Environments

Edwin Fong, Lancelot F. James, Juho Lee · 2025

Read Learning from Neighbors with PHIBP: Predicting Infectious Disease Dynamics in Data-Sparse Environments by Edwin Fong, Lancelot F. James, Juho Lee (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-21005.txt.

Introduction

Edwin Fong, Lancelot James, and Juho Lee tackle epidemic forecasting where the data is mostly zeros. Regions that have never reported a case still need outbreak predictions, and relative-rate methods wobble badly around zero counts. Their framework, the Poisson Hierarchical Indian Buffet Process, is built on absolute abundance and had already proven itself on sparse microbiome and ecological counts. Applied to infectious disease, it borrows statistical strength from related regions instead of treating each silent region as a blank.

Experiments on disease data produce coherent forecasts even for areas with no cases so far, and the setup supports diversity contrasts that ecologists know well. For public-health modelers, the chapter supplies both the computing tools and evidence that sound borrowing across neighbors beats patching zeros.

Catalog record

CREATOREdwin Fong, Lancelot F. James, Juho Lee
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE30 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2512-21005

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning from Neighbors with PHIBP: Predicting Infectious Disease Dynamics in Data-Sparse Environments Edwin Fong, Lancelot F. James, Juho Lee Abstract Modeling sparse count data, which arise across numerous scientific fields, presents significant statistical challenges. This chapter addresses these challenges in the context of infectious disease prediction, with a focus on predicting outbreaks in geographic regions that have historically reported zero cases. To this end, we present the detailed computational framework and experimental application of the Poisson Hierarchical Indian Buffet Process (PHIBP) [ 5 ] , a Bayesian machine learning model with demonstrated success in handling sparse count data in microbiome and ecological studies. The PHIBP’s architecture, grounded in the concept of absolute abundance, systematically borrows statistical strength from related regions and circumvents the known sensitivities of relative-rate methods to zero counts. Through a series of experiments on infectious disease data, we show that this principled approach provides a robust foundation for generating coherent predictive distributions and for the effective use of comparative measures such as alpha and beta diversity. The chapter’s emphasis on algorithmic implementation and experimental results confirms that this unified framework delivers both accurate outbreak predictions and meaningful epidemiological insights in data-sparse environments. , , and ?? Department of Statistics and Actuarial Science, HKU, ?? ?? Department of ISOM, HKUST, ?? ?? The Graduate School of AI, KAIST, ?? 1 Introduction Predicting disease prevalence in regions that have not yet reported cases—despite clear outbreaks in neighboring areas—is a problem most people now recognize from lived experience: when will it reach us, and how severe might it be? Turning this intuitive concern into reliable forecasts is a statistically challenging task. Historically, one can relate this to classes of count - based models for both realized abundance (counts) and prediction of unseen numbers and types of species in the classic work of [ 2 ] . There Fisher, in answer to Corbett’s question about the expected number of distinct species of butterflies to be seen in a future sample, developed the logarithmic series distribution to describe the prevalence of individually observed species. This then translates into the remarkable work of [ 6 ] , which empirically verifies the Negative Binomial distribution as a viable model for soil microorganism counts in the form of a mixed Poisson variable: the framework in [ 6 ] provides an empirical justification for a model in which the number of colonies is distributed according to a Poisson distribution, while the bacterial counts within each colony independently follow the logarithmic series distribution introduced by [ 2 ] . Naturally, while many core elements of those works translate to modern datasets, they do not capture the notion of sharing information across groups, nor matters of count sparsity. In particular, while the prevalence of zeros is common in modern datasets, we further highlight the notion of sparse co-occurrence, where samples across groups do not share many common species. These challenges, along with modeling complex multivariate count distributions, are all addressed by the Poisson Hierarchical Indian Buffet Process (PHIBP) [ 5 ] as applied to complex microbiome sampling models. Here, we focus on its ability to pool information across groups to handle zero observations, leading to credible predictive and inferential modeling for infectious disease prediction. As the theory of PHIBP has been detailed previously, our focus is on methodology and computational inference within the present context of disease prediction. 2 The PHIBP model We consider J related regions (e.g. counties), j\in[J] , each with M_{j} replicated count samples (e.g. years), i\in[M_{j}] . A sample has the form Z^{(i)}_{j}\;=\;\sum_{l\geq 1}N^{(i)}_{j,l}\,\delta_{Y_{l}}, where Y_{l} indexes a global catalogue of disease types and N^{(i)}_{j,l}\in\{0,1,2,\dots\} is the count of disease Y_{l} in sample i from region j . The PHIBP specifies a hierarchical prior on the latent mean abundance rates that generate these counts. At the top level, we draw a global completely random measure B_{0}\;=\;\sum_{l\geq 1}\lambda_{l}\,\delta_{Y_{l}}\;\sim\;\mathrm{CRM}(\tau_{0},F_{0}), where \tau_{0} is a Lévy measure on (0,\infty) and F_{0} a non-atomic probability measure on a Polish space \Omega of disease labels. The jumps (\lambda_{l}) are interpreted as global mean rates of the diseases (Y_{l}) across all regions. Given B_{0} , each region j has a local CRM B_{j}\mid B_{0}\;\sim\;\mathrm{CRM}(\tau_{j},B_{0}),\qquad j\in[J], with Lévy density \tau_{j} on (0,\infty) . Conditionally on B_{0}=\sum_{l}\lambda_{l}\delta_{Y_{l}} we may write B_{j}\;\stackrel{{\scriptstyle d}}{{=}}\;\sum_{l\geq 1}\sigma_{j,l}(\lambda_{l})\,\delta_{Y_{l}},\qquad\sigma_{j,l}(\lambda_{l})\stackrel{{\scriptstyle d}}{{=}}\sigma_{j}(\lambda_{l}), where (\sigma_{j}(t):t\geq 0) is a subordinator with Lévy density \tau_{j}, satisfying for s<t, \sigma_{j}(t)-\sigma_{j}(s)\overset{d}{=}\sigma_{j}(t-s) independent of \sigma_{j}(s) . Thus \lambda_{l} is the global rate of disease Y_{l} , and \sigma_{j,l}(\lambda_{l}) is its local mean rate in region j . Throughout, as in [ 5 ] , we denote by \psi_{j}(\cdot) and \Psi_{0}(\cdot) the Laplace exponents associated with the Lévy measures \tau_{j} and \tau_{0} , respectively. In particular, for each j and exposure \gamma_{j}>0 , \mathbb{E}[e^{-\gamma_{j}\sigma_{j}(\lambda)}]=e^{-\lambda\psi_{j}(\gamma_{j})} , where \psi_{j}(\gamma_{j})\;=\;\int_{(0,\infty)}\big(1-e^{-\gamma_{j}s}\big)\,\tau_{j}(ds), and similarly \Psi_{0}(\gamma_{0})\;=\;\int_{(0,\infty)}\big(1-e^{-\gamma_{0}\lambda}\big)\,\tau_{0}(d\lambda),\quad\gamma_{0}\geq 0. where we assume that \int_{0}^{\infty}\min(s,1)\tau_{j}(s)ds<\infty ensuring the finiteness of B_{j}(\Omega) for j=0,\ldots,J. We will only use these via evaluations such as \psi_{j}\!\big(\sum_{i=1}^{M_{j}}\gamma_{i,j}\big) and \Psi_{0}\big(\sum_{j=1}^{J}\psi_{j}(\cdot)\big) , exactly as in the PHIBP construction of [ 5 ] . Furthermore setting \psi^{(c)}_{j}(\gamma_{j})=\int_{0}^{\infty}s^{c}e^{-\gamma_{j}s}\tau_{j}(s)ds we have the mixed truncated Poisson distributions with law denoted as \mathrm{MtP}(\tau_{j},\gamma_{j}) and probability mass function for c=1,2,\ldots, given by \frac{\gamma^{c}_{j}\psi^{(c)}_{j}(\gamma_{j})}{\psi_{j}(\gamma_{j})c_{!}} The PHIBP (Z^{(i)}_{j},i\in[M_{j},j\in[J]) is specified as a mixed Poisson processes driven by the local CRMs. For each j and i\in[M_{j}] , Z^{(i)}_{j}\,\big|\,B_{j}\;\stackrel{{\scriptstyle\mathrm{ind}}}{{\sim}}\;\mathrm{PoiP}(\gamma_{i,j}B_{j}), (2.1) where \gamma_{i,j}>0 is a sample-specific exposure. Using the atomic form of B_{j} , this means Z^{(i)}_{j}\;\stackrel{{\scriptstyle d}}{{=}}\;\sum_{l\geq 1}\mathscr{P}^{(i)}_{j,l}\big(\gamma_{i,j}\sigma_{j,l}(\lambda_{l})\big)\,\delta_{Y_{l}}, (2.2) with conditionally independent Poisson counts \mathscr{P}^{(i)}_{j,l}\big(\gamma_{i,j}\sigma_{j,l}(\lambda_{l})\big)\;\sim\;\mathrm{Poisson}\!\big(\gamma_{i,j}\sigma_{j,l}(\lambda_{l})\big). This is the basic mixed Poisson representation: each observed count is Poisson with a random intensity given by the exposure \gamma_{i,j} times the local rate \sigma_{j,l}(\lambda_{l}) . Summing over samples within region j gives \sum_{i=1}^{M_{j}}Z^{(i)}_{j}\,\bigg|\,B_{j}\;\sim\;\mathrm{PoiP}\!\Big(\Big(\sum_{i=1}^{M_{j}}\gamma_{i,j}\Big)B_{j}\Big) and induces the usual quantity \psi_{j}\!\Big(\sum_{i=1}^{M_{j}}\gamma_{i,j}\Big)\;=\;\int_{(0,\infty)}\Big(1-e^{-s\sum_{i=1}^{M_{j}}\gamma_{i,j}}\Big)\,\tau_{j}(ds), which governs the rate of species (atoms) observed in region j after aggregating over its M_{j} samples. Writing N_{j,l}:=\sum_{i=1}^{M_{j}}N^{(i)}_{j,l} for the total count of disease Y_{l} in region j , we have N_{j,l}\,\big|\,(\sigma_{j,l}(\lambda_{l}))_{j,l}\;\sim\;\mathrm{Poisson}\!\Big(\Big(\sum_{i=1}^{M_{j}}\gamma_{i,j}\Big)\sigma_{j,l}(\lambda_{l})\Big), independently over (j,l) . The key result (Theorem 3.1 of [ 5 ] ) shows that the joint law of the summed processes \Big(\sum_{i=1}^{M_{j}}Z^{(i)}_{j},\;j\in[J]\Big)\;\overset{d}{=}\;\Big(\sum_{\ell=1}^{\varphi}\tilde{N}_{j,\ell}\,\delta_{\tilde{Y}_{\ell}},\;j\in[J]\Big)\;\overset{d}{:=}\;\Big(\sum_{\ell=1}^{\varphi}\big[\sum_{k=1}^{X_{j,\ell}}C_{j,k,\ell}\big]\delta_{\tilde{Y}_{\ell}},\;j\in[J]\Big). (2.3) admits an exact compound Poisson representation in terms of a random number of latent (sub)-species level clusters and their counts. Here \varphi is the random number of distinct species that appear across all regions, and has a Poisson distribution with mean \Psi_{0}\big(\sum_{j=1}^{J}\psi_{j}(\sum_{i=1}^{M_{j}}\gamma_{i,j})\big) . For each observed species \ell , there is a global posterior mean rate H_{\ell}>0 and a corresponding total number of OTU-level clusters X_{\ell}:=\sum_{j=1}^{J}X_{j,\ell} ; jointly, (H_{\ell},X_{\ell}) has the mixed Poisson–MtP structure of [ 5 ] , with X_{\ell}\sim\mathrm{MtP}\big(\tau_{0},\sum_{j=1}^{J}\psi_{j}(\sum_{i=1}^{M_{j}}\gamma_{i,j})\big) and H_{\ell} conditionally distributed with density proportional to \lambda^{X_{\ell}}\exp\!\{-\lambda\sum_{j=1}^{J}\psi_{j}(\sum_{i=1}^{M_{j}}\gamma_{i,j})\}\tau_{0}(\lambda) . Given X_{\ell}=x_{\ell} , the allocation of OTU clusters across regions is multinomial, (X_{1,\ell},\dots,X_{J,\ell})\mid X_{\ell}=x_{\ell}\sim\mathrm{Multinomial}(x_{\ell};q_{1},\dots,q_{J}) , with q_{j}\propto\psi_{j}\big(\sum_{i=1}^{M_{j}}\gamma_{i,j}\big) . Finally, given these allocations and the local Lévy measures \tau_{j} , the OTU-level cluster sizes C_{j,k,\ell} are independent mixed truncated Poisson variables C_{j,k,\ell}\sim\mathrm{MtP}\big(\tau_{j},\sum_{i=1}^{M_{j}}\gamma_{i,j}\big) , so that each regional total \tilde{N}_{j,\ell}=\sum_{k=1}^{X_{j,\ell}}C_{j,k,\ell} is a sum of i.i.d. MtP components driven by the global rate H_{\ell} . 2.1 Predicting the unseen and alpha/beta diversities We now recount some of the novel developments in [ 5 ] that will figure prominently in our analysis. We will use this compound Poisson form, together with the posterior local rates \tilde{\sigma}_{j,l}(H_{l}) , to define and compute our Bayesian

Read the complete text