Integration of Large Language Models and Traditional Deep Learning for Social Determinants of Health Prediction

Paul Landes, Jimeng Sun, Adam Cross · 2025

Read Integration of Large Language Models and Traditional Deep Learning for Social Determinants of Health Prediction by Paul Landes, Jimeng Sun, Adam Cross (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-04655.txt.

Introduction

Paul Landes, Jimeng Sun, and Adam Cross combine large language models with leaner deep learning to extract social determinants of health from clinical notes in a 2025 paper. Income, housing, and personal circumstances shape health, so coding them helps doctors judge risk. The team compare the two model families on a public data set. Their combined approach beat a prior reference score by a wide margin.

The trick is division of labor: the language model supplies precision, and the smaller model supplies speed, cutting execution time many times over by skipping heavy processing where it is not needed. Traditional models trained with added synthetic data even beat the language models on some tests. The authors present the blend as the next practical step for flagging at risk patients automatically.

Catalog record

CREATORPaul Landes, Jimeng Sun, Adam Cross
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE43 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-04655

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Integration of Large Language Models and Traditional Deep Learning for Social Determinants of Health Prediction Paul Landes, Jimeng Sun, Adam Cross Abstract Social Determinants of Health are economic, social and personal circumstances that affect or influence an individual’s health status. Social Determinants of Healths have shown to be correlated to wellness outcomes, and therefore, are useful to physicians in diagnosing diseases and in decision-making. In this work, we automatically extract Social Determinants of Healths from clinical text using traditional deep learning and Large Language Models to find the advantages and disadvantages of each on an existing publicly available dataset. Our models outperform a previous reference point on a multilabel Social Determinants of Health classification by 10 points, and we present a method and model to drastically speed up classification (12X execution time) by eliminating expensive Large Language Model processing. The method we present combines a more nimble and efficient solution that leverages the power of the Large Language Model for precision and traditional deep learning methods for efficiency. We also show highly performant results on a dataset supplemented with synthetic data and several traditional deep learning models that outperform Large Language Models. Our models and methods offer the next iteration of automatic prediction of Social Determinants of Healths that impact at-risk patients. 1 Introduction Social Determinants of Health greatly affect health, well-being and quality of life. Safe housing, job opportunities, discrimination, and environmental factors are just a few examples Social Determinants of Health types. Adverse Social Determinants of Healths have immediate and profound affects on patient’s health, such as air quality’s negative affect on one’s respiratory system or access to healthcare for those without income stability. The impact of Social Determinants of Healths on health outcomes have been shown to be helpful in prediction of diabetes mellitus [ 1 ] , recurring diabetic keto-acidosis [ 2 ] , and prolonged hospital stays [ 3 ] . Although electronic health record systems have incorporated Social Determinants of Healths as structured data, many systems have not, or represent the data with highly varying formats [ 4 , 5 , 6 ] . To address these challenges, many have turned natural language processing methods to automatically extract Social Determinants of Healths from clinical text. The current state-of-the-art for this type of extraction incorporates Large Language Models in either a few-shot learning or supervised-fine tuning setting [ 7 , 8 ] . Dataset Model Size Setting wF1 wP wR mF1 mP mR MF1 MP MR MIMIC-III [ 7 ] [ 7 ] 11B Fine-tune - - - - - - 0.57 - - Llama 3.1 Instruct 8B Few-shot 0.84 0.94 0.78 0.74 0.71 0.78 0.32 0.28 0.69 Llama 3.3 Instruct 70B Few-shot 0.92 0.95 0.91 0.88 0.86 0.91 0.47 0.39 0.85 Llama 3.1 Instruct 8B Fine-tune 0.95 0.95 0.95 0.95 0.96 0.95 0.67 0.65 0.78 Llama 3.3 Instruct 70B Fine-tune 0.95 0.95 0.96 0.96 0.96 0.96 0.64 0.70 0.72 RoBERTa Base 110M Fine-tune 0.96 0.96 0.96 0.97 0.97 0.96 0.53 0.60 0.49 Two-step 8.11B Fine-tune 0.95 0.96 0.96 0.96 0.97 0.96 0.62 0.76 0.59 MIMIC-III + [ 7 ] [ 7 ] 11B Fine-tune - - - - - - 0.55 - - Synthetic Llama 3.1 Instruct 8B Few-shot 0.81 0.89 0.78 0.74 0.71 0.78 0.53 0.45 0.78 Llama 3.3 Instruct 70B Few-shot 0.90 0.92 0.91 0.88 0.85 0.91 0.70 0.61 0.90 Llama 3.1 Instruct 8B Fine-tune 0.94 0.95 0.94 0.95 0.95 0.94 0.78 0.86 0.74 Llama 3.3 Instruct 70B Fine-tune 0.96 0.96 0.96 0.96 0.95 0.96 0.86 0.82 0.90 RoBERTa Base 110M Fine-tune 0.96 0.97 0.96 0.97 0.97 0.96 0.88 0.91 0.86 Table 1: Multilabel model performance by average. The performance for the MIMIC-III and Amended datasets with the top performing weighted and macro averages bolded . The traditional deep learning approach is given as the result using the RoBERTa Base model. The metrics includes include ( w )eighted, ( m )icro and ( M )acro metric averages. Dataset Model Size Setting No SDoH Employ Hous Parent Relation Sup Tran MIMIC-III [ 7 ] [ 7 ] 11B Fine-tune 0.98 0.65 0.00 0.63 0.91 0.32 0.50 Llama 3.1 Instruct 8B Few-shot 0.88 0.41 0.01 0.26 0.46 0.23 0.03 Llama 3.3 Instruct 70B Few-shot 0.95 0.77 0.06 0.33 0.77 0.24 0.19 Llama 3.1 Instruct 8B Fine-tune 0.98 0.67 1.00 0.53 0.86 0.14 0.50 Llama 3.3 Instruct 70B Fine-tune 0.98 0.35 0.40 0.80 0.88 0.08 1.00 RoBERTa Base 110M Fine-tune 0.98 0.57 0.00 0.89 0.85 0.41 0.00 Two-step 8.11B Fine-tune 0.98 0.64 1.00 0.67 0.88 0.16 0.00 MIMIC-III + [ 7 ] [ 7 ] 11B Fine-tune 0.98 0.69 0.24 0.44 0.91 0.33 0.24 Synthetic Llama 3.1 Instruct 8B Few-shot 0.87 0.58 0.14 0.57 0.62 0.38 0.53 Llama 3.3 Instruct 70B Few-shot 0.95 0.83 0.49 0.64 0.85 0.36 0.76 Llama 3.1 Instruct 8B Fine-tune 0.98 0.70 0.80 0.68 0.94 0.53 0.81 Llama 3.3 Instruct 70B Fine-tune 0.98 0.95 0.97 0.80 0.94 0.59 0.80 RoBERTa Base 110M Fine-tune 0.99 0.94 0.92 0.80 0.93 0.68 0.92 Table 2: Multilabel model performance by label. The performance each multi-label classification on the MIMIC-III and Amended datasets with the top performing metrics bolded . Predicted labels are ( Employ )ment, ( Hou )sing, Parent, ( Relation )ship, ( Sup )port, and ( Tran )sportation. The traditional deep learning approach is given as the result using the RoBERTa Base model. We build on the work of [ 7 ] [ 7 ] by training a variety of similar models on their publicly released datasets 1 1 1 We can not compare directly as [ 7 ] [ 7 ] did not release their model, training dataset or prompts. In their work, the MIMIC-III and synthetic datasets were used for testing only. (see Table 7 ). The first dataset is subset of the Medical Information Mart for Intensive Care III corpus [ 9 ] , a large freely accessible hospital database of ICU data from the Beth Israel Deaconess Medical Center in Boston, Massachusetts. This dataset consists of 5,355 sentences taken from Medical Information Mart for Intensive Care III clinical notes that were annotated for zero or more Social Determinants of Healths. A synthetic dataset generated from Large Language Models was also released, containing one or more Social Determinants of Healths, with 588 sentences annotated with at least one Social Determinants of Health. The synthetic dataset was intended to provide a larger training set to boost results on the test dataset (Medical Information Mart for Intensive Care III). Our experiments test Llama models [ 10 , 11 ] for multilabel classification, and encoder-only BERT [ 12 ] models for both multilabel and binary classification. We experiment with several feature combinations in our traditional models and our Large Language Model models with few-shot and supervised fine-tuning settings. After error and performance analysis, we integrated both Large Language Models and traditional deep learning natural language processing models to exploit the best of both worlds. Our binary traditional deep learning models performed inference up to 73 times faster than our slowest Large Language Model (see Table 6 ). Based on this observation, we adopted a new two-step model that predicts sentences with at least one Social Determinants of Health. For those that do, we use

Read the complete text