General Demographic Foundation Models for Enhancing Predictive Performance Across Diseases

Li-Chin Chen, Ji-Tian Sheu, Yuh-Jue Chuang · 2025

Read General Demographic Foundation Models for Enhancing Predictive Performance Across Diseases by Li-Chin Chen, Ji-Tian Sheu, Yuh-Jue Chuang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07330.txt.

Introduction

Li-Chin Chen's 2025 paper, written with 2 co-authors, builds a pre-trained model that makes age and sex more useful in medical prediction. Foundation models for health care must work across many diseases and patient groups while staying simple enough to deploy. Age and sex are recorded almost everywhere, yet standard models may underuse what they reveal about a patient's state.

The General Demographic Pre-trained model, or GDP, learns rich summaries of a patient from age and sex alone. Adding those summaries beside the raw features improves prediction across several diseases and regions with distinct patient mixes. The team tuned how features are encoded and how visits are ordered, then compared GDP against leading tabular foundation models. Gains held up across tasks, which suggests even two basic fields carry untapped signal for clinical models.

Catalog record

CREATORLi-Chin Chen, Ji-Tian Sheu, Yuh-Jue Chuang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE27 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-07330

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

General Demographic Foundation Models for Enhancing Predictive Performance Across Diseases Li-Chin Chen, Ji-Tian Sheu, Yuh-Jue Chuang Abstract Demographic attributes are universally present in electronic health records and serve as vital predictors in clinical risk stratification and treatment decisions. Despite their significance, these attributes are often relegated to auxiliary roles in model design, with limited attention has been given to learning their representations. This study proposes a General Demographic Pre-trained (GDP) model as a foundational representation framework tailored to age and gender. The model is pre-trained and evaluated using datasets with diverse diseases and population compositions from different geographic regions. The GDP architecture explores combinations of ordering strategies and encoding methods to transform tabular demographic inputs into latent embeddings. Experimental results demonstrate that sequential ordering substantially improves model performance in discrimination, calibration, and the corresponding information gain at each decision tree split, particularly in diseases where age and gender contribute significantly to risk stratification. Even in datasets where demographic attributes hold relatively low predictive value, GDP enhances the representational importance, increasing their influence in downstream gradient boosting models. The findings suggest that foundational models for tabular demographic attributes can generalize across tasks and populations, offering a promising direction for improving predictive performance in healthcare applications. Keywords Foundational Model \cdot Demographic Attribute \cdot Representation Learning \cdot Model Transferability 1 Introduction Electronic Health Records (EHRs) provide a rich, chronologically ordered record of patient care, encompassing a broad spectrum of medical events. In their tabular form, EHR datasets store diverse attributes for each encounter, including diagnoses, procedures, medications, laboratory results, often encoded with standardized clinical terminologies such as ICD, LOINC, and SNOMED, etc. duan2024deep ; li2022hiBEHRT . The proliferation of EHR adoption has yielded an invaluable material for training sophisticated healthcare AI systems. Mainstream deep learning research, however, has gravitated toward homogeneous data modalities, such as computer vision, NLP, and speech, while tabular data remains underexplored borisov2022deep ; somvanshi2024survey . This disparity constrains the advancement of foundational models in tabular data. Foundational models, by definition, are pre-trained on large, heterogeneous datasets and capable of adaptation across a wide range of tasks he2024foundation ; bommasani2021opportunities ; moor2023foundation ; 2024ehrshot_followup ; borisov2022deep . Within healthcare, foundational model applications have predominantly centered on language, imaging, bioinformatics (e.g., genomic and proteomic sequences), and multimodal fusion he2024foundation ; azad2023foundational ; khan2025comprehensive . By contrast, tabular data, arguably the most prevalent data form in healthcare, has been comparatively neglected. The construction of foundational models for tabular data presents notable challenges. Unlike images or text, tabular data is inherently heterogeneous, combining dense numerical variables with sparse categorical features. Inter-feature correlations are often weak and irregular, lacking the spatial or semantic structure found in other modalities, which complicates the extraction of meaningful relationships without spatial priors borisov2022deep ; somvanshi2024survey . Demographic attributes (e.g., age, gender, race) are among the most fundamental and readily available patient characteristics. Despite their ubiquity, they are frequently treated as auxiliary features rather than as core representational inputs. This study focuses on constructing foundational representations for demographic attributes, with a particular emphasis on age and gender. 1.1 Representation Learning for Demographic Attributes A key strength of deep representation learning lies in its capacity to obviate manual feature engineering by learning rich, hierarchical representations in an end-to-end fashion borisov2022deep ; lecun2015deep ; bengio2013representation ; schmidhuber2015deep . Trained on large-scale datasets, deep neural networks can automatically derive high-level abstractions from raw inputs, with intermediate layers functioning as sophisticated feature extractors borisov2022deep ; fallahpour2024ehrmamba ; rasmy2021medBERT ; yuanyuan2025scoping . Such learned embeddings have consistently been shown to enhance downstream predictive performance borisov2022deep ; MyGLP ; yoon2020vime . While deep learning excels on homogeneous data (e.g., images, audio, text) that possess strong spatial or sequential structures borisov2022deep ; ghosh2019investigating , the encoding of EHR data often involves arranging patient visits into sequences 2021MEDBERT ; yang2023transformehr ; hur2023genhpf ; yuanyuan2025scoping . For instance, Yang et al. yang2023transformehr incorporated demographic and ICD code embeddings, summing visit embeddings (which preserved temporal order), temporal embeddings (encoding visit dates or inter-visit intervals via sinusoidal positional encodings), and code/demographic embeddings to form the model input. Wornow et al. wornow2023ehrshot and Hur et al. hur2023genhpf both transformed all medical events into natural language descriptions and then tokenized them into embeddings via a language model encoder, whereas Wornow et al. converted medical codes into discretized value ranges and Hur et al. expressed them as direct descriptions. Some works Fourier-transform age into sinusoidal sequences fallahpour2024ehrmamba ; kazemi2019time2vec and sum with other concept embedding, whereas others omit demographic attributes 2021MEDBERT . In prior research, demographic information has generally been incorporated as auxiliary context rather than serving as the principal focus of encoder architecture design. Nevertheless, demographic attributes are highly standardized, readily obtainable, and inherently informative, and thus merit dedicated representational modeling. Within this category, age functions as a critical determinant, providing signals of biological vulnerability, diagnostic framing, therapeutic constraints, risk stratification, and eligibility for age-specific screening jackson2004burden ; mandelblatt2021applying ; american2012patient . Similarly, gender or sex constitutes a fundamental axis along which patients may exhibit differing physiological responses to a wide range of diseases oertelt2009gender ; courchesne2023gender ; moores2023sex . To address this gap, the present study introduces a General Demographic Pre-trained (GDP) model, which is conceptualized as a foundational model centered on age and gender, aimed at enhancing predictive performance across multiple disease domains. 2 Methods Figure 1: General demographical pre-trained model training and validation flow This study aims to develop a GDP model, conceptualized as a foundational model designed to enhance the predictive utility of demographic attributes. The overall workflow is illustrated in Figure 1 . The process begins with training the GDP model solely on age and gender information (Figure 1 a), followed by applying the resulting embeddings to three distinct downstream predictive tasks (Figure 1 b). To guide the embedding learning process, the GDP model was trained to predict the Charlson Comorbidity Index (CCI) charlson1987new at each patient visit. The CCI is a widely adopted measure for assessing patient mortality risk and disease severity based on comorbid conditions, and thus serves as a robust proxy for generating embeddings that capture clinically meaningful health status representations. The study further evaluated three distinct encoding strategies and two input ordering schemes. 2.1 Data Encoding and Sequential Ordering Evaluation Three encoding strategies were examined. First is the traditional encoding ( \emph{trad} ), gender ( g ) was represented using one-hot encoding, where g\in\{0,1\} , and age ( x ) was transformed with the natural logarithm x_{e}=\log(1+x) , where x\in\mathbb{R}^{n} and e denotes the resulting embedding vector. Second is the positional encoding ( \emph{PE} ) approach embedded age information using a sinusoidal positional encoding scheme and further differentiated it by adding zeros or ones for each gender. This is expressed as: PE_{(pos,2i)}=\sin\left(\frac{pos}{10000^{2i/d_{model}}}\right)+g_{i}\\ (1) PE_{(pos,2i+1)}=\cos\left(\frac{pos}{10000^{2i/d_{model}}}\right)+g_{i} (2) , where pos is the position index, i is the dimension index, g_{i}\in\{0,1\} , and g\in\mathbb{R}^{d_{\text{model}}} . Finally, in the text-based semantic encoding ( \emph{txt} ), demographic information was first expressed as short descriptive text strings (e.g., Male, 75 years old) and subsequently converted into embeddings using the encoder of an open language model: E_{w_{i}}=f_{encoder}([w_{1},\dots,w_{i}]) (3) , w_{i} denotes the i^{\text{th}} token, E_{w_{i}}\in\mathbb{R}^{d} is its embedding vector, and d represents the embedding dimension. The encoder used was the

Read the complete text