Infinite hierarchical contrastive clustering for personal digital envirotyping

Ya-Yun Huang, Joseph McClernon, Jason A. Oliver, Matthew M. Engelhard · 2025

Read Infinite hierarchical contrastive clustering for personal digital envirotyping by Ya-Yun Huang, Joseph McClernon, Jason A. Oliver, Matthew M. Engelhard (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-15022.txt.

Introduction

Ya-Yun Huang and three co-authors present a 2025 method for sorting photos of daily life. People in health studies take phone photos of the places they pass through each day. Grouping those photos by place can link daily settings to health and habits, but the groups are not known in advance.

Their method builds clusters in layers, so single places sit inside wider types of places. It does not need a set number of groups at the start. Tests show the method finds clear personal places and sorts them into types that make sense. The authors then show how the groups can be tied to health outcomes for each person.

Catalog record

CREATORYa-Yun Huang, Joseph McClernon, Jason A. Oliver, Matthew M. Engelhard
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE31 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-15022

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Infinite hierarchical contrastive clustering for personal digital envirotyping Ya-Yun Huang, Joseph McClernon, Jason A. Oliver, Matthew M. Engelhard Abstract Daily environments have profound influence on our health and behavior. Recent work has shown that digital envirotyping , where computer vision is applied to images of daily environments taken during ecological momentary assessment (EMA), can be used to identify meaningful relationships between environmental features and health outcomes of interest. To systematically study such effects on an individual level, it is helpful to group images into distinct environments encountered in an individual’s daily life; these may then be analyzed, further grouped into related environments with similar features, and linked to health outcomes. Here we introduce infinite hierarchical contrastive clustering to address this challenge. Building on the established contrastive clustering framework, our method a) allows an arbitrary number of clusters without requiring the full Dirichlet Process machinery by placing a stick-breaking prior on predicted cluster probabilities; and b) encourages distinct environments to form well-defined sub-clusters within each cluster of related environments by incorporating a participant-specific prediction loss. Our experiments show that our model effectively identifies distinct personal environments and groups these environments into meaningful environment types. We then illustrate how the resulting clusters can be linked to various health outcomes, highlighting the potential of our approach to advance the envirotyping paradigm. † † workshop: Machine Learning for Health (ML4H) 2024 keywords contrastive learning, envirotyping, EMA, ecological momentary assessment Data and Code Availability We use two sources of daily environment data in our work. The first source comprises images collected from our study team to evaluate the performance of our method and share the results publicly. The second source includes images and associated health outcomes collected from a larger group of participants through a photo EMA process. This data was used to explore the relationship between daily environments and health outcomes. Due to the lack of written consent from participants for public data sharing, these data cannot be made available for public access. All code needed to define and train our model may be found at ih-cc-envirotyping . Institutional Review Board (IRB) Recruitment and all study procedures were approved by the Duke University Health System Institutional Review Board, and written consent was obtained from all participants. 1 Introduction The influence of daily environments on human health and behavior has long been recognized ( Moser and Uzzell 2003 ) . For example, neighborhood aesthetics and access to green spaces have been linked to depressive moods ( Rautio et al. 2018 ; Tabet et al. 2017 ) , and there are established links between daily environments and obesity, psychosocial health, and more ( Kirk et al. 2010 ; Amerio et al. 2020 ) . Home environments are also important but less studied; for example, personal smoking environments have been shown to elicit the urge to smoke and increase smoking behavior ( McClernon et al. 2016 ) . These findings underscore the importance of characterizing individuals’ daily environments, which we call envirotyping , and analyzing relationships between individual envirotypes and health and behavioral outcomes. Current technologies such as mobile phones allow us to sample personal daily environments together with other features such as GPS on an ongoing basis at almost no cost and minimal user burden. The resulting images can then be processed via computer vision (CV) models to analyze and predict a variety of health and behavioral outcomes ( Engelhard et al. 2019 ) . However, rather than connecting environmental images and features directly to outcomes, we would like to identify distinct environments that an individual encounters in their daily life, which requires an automated method to accurately group multiple images of the same environment. This would then allow us to analyze each environment, summarize the health outcomes associated with it, and approximate the amount of time the individual spends there. Each of these is important both to understand the influences of environments on health, and to develop environment-aware intervention strategies. Many existing approaches might be considered for this task, but ultimately fall short. Pre-trained image classification models can identify common environment types, but fail to identify distinct environments or adapt to the broad range of settings encountered in daily life. GPS can help differentiate between locations that are far apart, but is not sufficiently precise to distinguish between distinct environments within the home. Most promising are image clustering methods, but to our knowledge they have not been applied to identify distinct personal environments, and consequently are not optimized for this purpose. Here we propose a scalable, effective CV method designed to facilitate digital envirotyping by a) grouping images into distinct environments, and b) grouping multiple distinct environments into clusters with similar environmental features ( i.e. , environment types). Our approach, which builds on the contrastive clustering (CC) framework proposed by Li et al. 2021 , is motivated by the observation that the data augmentations applied in contrastive learning mirror the variability encountered during repeated sampling of environmental images. For example, cropping and brightness adjustment simulate changes in vantage point and lighting conditions, respectively. However, applying the CC framework directly is not optimal, as it has two key limitations in this context. First, CC is limited to a fixed number of clusters, which poses a challenge when the number of distinct environments and environment types is unknown. To address this limitation, we propose an infinite contrastive clustering model that does not require us to fix the number of clusters a priori . Whereas the infinite clustering problem has previously been solved by placing a Dirichlet Process (DP) prior on latent features ( Lee et al. 2020 ) , we show that within the contrastive clustering framework, a similar effect can be achieved without the full DP by placing a stick-breaking prior on predicted cluster probabilities. Second, CC provides only one level of clustering, whereas our goal is not only to identify distinct environments, but also to define groups of environments with similar features. To address this, we associate clusters with the top level in this hierarchy, i.e. , environment types, but incorporate an additional supervisory signal that encourages distinct environments to form well-defined sub-clusters within each environment type. Specifically, we incorporate a participant-specific prediction head (PSH) that predicts the individual to whom each image belongs. Since all images of a given distinct environment belong to the same participant, this indirectly encourages formation of distinct sub-clusters and discourages the model from dividing them between multiple clusters. To illustrate the use of this method and quantify its effectiveness, we first apply our model to images of daily environments collected by our study team. This allows us to a) verify that images of the same environment are indeed assigned to the same cluster; and b) share the images publicly to illustrate the effectiveness of our approach. We then apply the model to a large photo-augmented EMA dataset designed to study the influence of daily environments on smoking and related outcomes. This provides a case study illustrating how the approach can help us understand relationships between environments and health. Overall, our goal is to facilitate envirotyping from repeated sampling of images of daily environments. To our knowledge, this is the first study to apply a contrastive learning approach to identify

Read the complete text