HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring
Xin Wang, Ting Dang, Xinyu Zhang, Vassilis Kostakos, Michael Witbrock, Hong Jia · 2025
Read HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring by Xin Wang, Ting Dang, Xinyu Zhang, Vassilis Kostakos, Michael Witbrock, Hong Jia (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-07260.txt.
Introduction
Xin Wang and 5 co-authors benchmark small language models for health monitoring on phones and wearables in a 2025 study. Small language models are compact versions of AI text models that can run on a device instead of in the cloud. Running locally protects privacy and cuts delay and memory use, but their health prediction skill was largely untested.
The team tests the small models with zero-shot, few-shot, and fine-tuning setups across health prediction tasks. The best fine-tuned models are then installed on real mobile devices to check speed and accuracy in practice. The small models match large models on many tasks while using far less power and keeping data on the device. Weak spots remain, notably uneven class sizes and few-shot cases where only a handful of examples guide the model.
Catalog record
| CREATOR | Xin Wang, Ting Dang, Xinyu Zhang, Vassilis Kostakos, Michael Witbrock, Hong Jia |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 47 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2509-07260
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
HealthSLM-Bench: Benchmarking Small Language Models for Mobile and Wearable Healthcare Monitoring Xin Wang, Ting Dang, Xinyu Zhang, Vassilis Kostakos, Michael Witbrock, Hong Jia Abstract Mobile and wearable healthcare monitoring play a vital role in facilitating timely interventions, managing chronic health conditions, and ultimately improving individuals’ quality of life. Previous studies on large language models (LLMs) have highlighted their impressive generalization abilities and effectiveness in healthcare prediction tasks. However, most LLM-based healthcare solutions are cloud-based, which raises significant privacy concerns and results in increased memory usage and latency. To address these challenges, there is growing interest in compact models, Small Language Models (SLMs), which are lightweight and designed to run locally and efficiently on mobile and wearable devices. Nevertheless, how well these models perform in healthcare prediction remains largely unexplored. We systematically evaluated SLMs on health prediction tasks using zero-shot, few-shot, and instruction fine-tuning approaches, and deployed the best performing fine-tuned SLMs on mobile devices to evaluate their real-world efficiency and predictive performance in practical healthcare scenarios. Our results show that SLMs can achieve performance comparable to LLMs while offering substantial gains in efficiency and privacy. However, challenges remain, particularly in handling class imbalance and few-shot scenarios. These findings highlight SLMs, though imperfect in their current form, as a promising solution for next-generation, privacy-preserving healthcare monitoring. 1 Introduction The proliferation of mobile and wearable devices, coupled with recent advances in deep learning, has significantly advanced the landscape of continuous health monitoring [ 1 , 2 , 3 , 4 ] . These technologies enable a range of real-time applications, from the detection of physiological anomalies [ 5 ] to the delivery of personalized interventions [ 6 ] . Meanwhile, large language models (LLMs) have demonstrated remarkable generalization in processing heterogeneous data and performing diverse downstream tasks [ 7 , 8 ] . Early studies indicate that LLM-based analysis can provide a deeper contextual interpretation of sensor data and enable more adaptive health monitoring systems compared to conventional approaches [ 9 ] . Despite this promise, major obstacles impede the practical deployment of LLM-driven wearable health solutions. Current approaches usually depend on cloud-based inference, necessitating data transmission to external servers, which raises concerns around user privacy, data security, and communication latency [ 10 , 11 , 12 , 13 ] . Alternatively, on-device deployment is hindered by severe resource constraints typical of mobile and wearable hardware, as well as the real-time requirements of health applications, rendering full-sized LLMs infeasible for timely inference. These challenges highlight a critical need for efficient, privacy-preserving techniques that achieve competitive performance with LLMs, while being suitable for deployment on resource-limited mobile and wearable devices. Small Language Models (SLMs) present a promising alternative by reducing memory consumption and facilitating deployment on mobile and wearable devices. On-device inference with SLMs not only lowers communication latency but also enhances the protection of sensitive personal data, while maintaining competitive performance on natural language processing tasks [ 14 , 15 , 16 , 17 ] . Nevertheless, their ability to interpret sensor data from mobile and wearable devices and accurately infer health conditions in real-world settings remains an open question. Although prior work [ 18 ] has demonstrated the feasibility of using SLMs on mobile devices to predict simple health status (e.g., fatigue, sleep quality), there is still a lack of comprehensive benchmarking that thoroughly evaluates SLMs for a wide range of health applications. To bridge this gap, we present a comprehensive benchmark, HealthSLM-Bench, which aims to evaluate a variety of state-of-the-art (SOTA) SLMs on a suit of health prediction tasks spanning three publicly available datasets. Our benchmark systematically assesses model performance using three evaluation protocols: zero-shot, few-shot, and instruction-based fine-tuning. To assess practical feasibility, we further deploy top-performing fine-tuned models on mobile devices and rigorously evaluate their on-device efficiency in terms of memory usage and inference latency. Experimental results demonstrate that SLMs can achieve comparable performance compared with SOTA healthcare LLMs across eight healthcare monitoring tasks, while substantially reducing memory and latency overheads. Our main contributions are as follows: • We introduce, HealthSLM-Bench, an extensive benchmark that systematically evaluates nine SOTA SLMs on eight health prediction tasks across three real-world mobile and wearable datasets. • We investigate various evaluation paradigms, including zero-shot, few-shot, and instruction-based fine-tuning, providing a comprehensive performance analysis under different adaptation scenarios. • We demonstrate the feasibility of deploying fine-tuned SLMs on resource-constrained mobile devices and quantify their efficiency in terms of real-world memory and latency footprints. 2 Related Work LLMs for health monitoring. With the rise of mobile and wearable devices, a variety of human-centered sensing signals can be continuously collected, enabling ongoing monitoring of human health in daily life. Recent studies have shown that the physical status data collected by mobile devices is strongly associated with health status [ 19 , 20 , 21 ] . Their work demonstrates how passive wearable sensor data can be effectively utilized to predict depression in adolescents using traditional ML models. However, these approaches, typically trained on specific datasets or tailored architectures, often struggle to generalize across heterogeneous tasks, and contexts [ 22 ] . LLMs, powered by their generalization capabilities, have shown great success in the healthcare domain. For example, Health-LLM [ 23 ] and MultiEEG-GPT [ 24 ] demonstrate the effectiveness of leveraging LLMs in healthcare monitoring through textual and physiological data. Instead of just deploying these models directly for healthcare applications, recent work has explored domain adaptation strategies such as few-shot prompting, instruction tuning, and domain-specific fine-tuning to improve performance on medical tasks [ 25 ] . Notably, PaLM2 [ 26 ] illustrates the benefits of combining diverse adaptation strategies (e.g. few-shot and fine-tuned) across medical datasets. Meanwhile, evaluations of GPT-4 highlight that SOTA LLMs may reduce the reliance on extensive adaptation, as they already demonstrate strong capacity for medical reasoning with limited supervision [ 27 ] . More recently, applied systems such as PhysioLLM [ 28 ] have integrated LLMs with wearable sensor data to provide personalized health insights, highlighting their adaptability across users and contexts. However, despite these advances, their computational overhead makes them impractical for privacy-sensitive, real-time mobile healthcare monitoring. Small Language Models. SLMs are defined as models that are smaller in scale relative to the widely recognised LLMs, typically comprising no more than 7 billion parameters [ 29 ] . Recent research has highlighted the efficiency and strong task performance of SLMs as lightweight alternatives to LLMs, particularly for deployment in resource-constrained environments [ 30 , 31 ] . For example, Phi-3-mini-4k-Instruct, developed by Microsoft [ 32 ] , contains 3.8 billion parameters and is trained on a curated blend of synthetic and high-quality public datasets, emphasizing reasoning capabilities. TinyLlama-1.1B [ 33 ] builds on Llama 2 through parameter reduction and subsequent fine-tuning using UltraChat, a broad synthetic dialogue dataset. Similarly, Google’s Gemma2-2B [ 34 ] , based on Gemini research, demonstrates robust results in text generation, summarization, and reasoning benchmarks. SmolLM-1.7B from HuggingFace [ 35 ] further diversifies training by leveraging synthetic educational materials and a breadth of domain samples, and Qwen2-1.5B [ 16 ] achieves SOTA performance in both coding and mathematics despite its small footprint. Meta’s Llama-3.2 series [ 36 ] continues this trend by releasing 1B and 3B parameter