FairWell : Fair Multimodal Self-Supervised Learning for Wellbeing Prediction
Jiaee Cheong, Abtin Mogharabin, Paul Liang, Hatice Gunes, Sinan Kalkan · 2025
Read FairWell : Fair Multimodal Self-Supervised Learning for Wellbeing Prediction by Jiaee Cheong, Abtin Mogharabin, Paul Liang, Hatice Gunes, Sinan Kalkan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-16748.txt.
Introduction
Jiaee Cheong, Abtin Mogharabin, Paul Liang, Hatice Gunes, and Sinan Kalkan make self-supervised learning fairer by feeding on messy data instead of fighting it, in a 2025 paper. Standard methods assume different views of a person carry the same message. Real health and behavior data break that: lengths vary and each channel holds unique signal. Forcing alignment then discards signal and bakes in bias.
Their framework, FairSSL, aligns segments from the same subject rather than across channels. A pooling trick handles uneven lengths, and the training goal balances spread, sameness, and decorrelation. Theory bounds the score gap between protected groups. On messy multimodal datasets, fairness rose without costing predictive power. The catalog title says FairWell; the paper itself names the method FairSSL.
Catalog record
| CREATOR | Jiaee Cheong, Abtin Mogharabin, Paul Liang, Hatice Gunes, Sinan Kalkan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 45 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2508-16748
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FairWell : Fair Multimodal Self-Supervised Learning for Wellbeing Prediction Jiaee Cheong, Abtin Mogharabin, Paul Liang, Hatice Gunes, Sinan Kalkan Abstract Early efforts on leveraging self-supervised learning (SSL) to improve machine learning (ML) fairness has proven promising. However, such an approach has yet to be explored within a multimodal context. Prior work has shown that, within a multimodal setting, different modalities contain modality-unique information that can complement information of other modalities. Leveraging on this, we propose a novel subject-level loss function to learn fairer representations via the following three mechanisms, adapting the variance-invariance-covariance regularization (VICReg) method: (i) the variance term, which reduces reliance on the protected attribute as a trivial solution; (ii) the invariance term, which ensures consistent predictions for similar individuals; and (iii) the covariance term, which minimizes correlational dependence on the protected attribute. Consequently, our loss function, coined as FairWell , aims to obtain subject-independent representations, enforcing fairness in multimodal prediction tasks. We evaluate our method on three challenging real-world heterogeneous healthcare datasets (i.e. D-Vlog, MIMIC and MODMA) which contain different modalities of varying length and different prediction tasks. Our findings indicate that our framework improves overall fairness performance with minimal reduction in classification performance and significantly improves on the performance-fairness Pareto frontier. Code and trained models will be made available at: https://is.gd/FAIRWELL 1 Introduction Figure 1: (a,b) Prior work have explored SSL for ML fairness in unimodal or tabular data settings. (c) FairWell addresses the challenges of non-tabular multimodal data in a subject-aware and modality-aware manner. Machine learning (ML) models are increasingly used in a wide-spectrum of healthcare settings ranging from epilepsy seizure detection ( Pontes et al. 2024 ) to pulmonary tuberculosis detection ( Chen et al. 2024 ) . Given the recent advances in multimodal machine learning ( Liang, Zadeh, and Morency 2024 ) , the natural extension of using multimodal ML for healthcare settings has proven to be promising ( Krones et al. 2025 ; Yildirim et al. 2024 ) . In concurrence, ML bias is becoming an increasing source of concern ( Ahmad et al. 2020 ; Cheong, Kalkan, and Gunes 2021 ; Cheong, Kalkan, and Gunes 2023a ) . However challenges in multimodal ML such as representation and alignment ( Liang, Zadeh, and Morency 2024 ) and the heterogeneity of healthcare data has made it difficult to advance ML fairness in multimodal healthcare settings. Within the context of multimodal ML for healthcare, learning relevant or good representations from temporally misaligned modalities of different feature and time length is often a challenge ( Chaptoukaev et al. 2024 ) (Gap 1) . Moreover, depending on the nature of the data modality, there are challenges such as data drift ( Pontes et al. 2024 ) , domain shift ( Jahanifar et al. 2025 ; Zhang et al. 2021 ) , multi-view redundancy ( Liang et al. 2023 ) , and data scarcity ( Bansal, Sharma, and Kathuria 2022 ) which are not adequately dealt with by existing models ( Rotalinti et al. 2022 ; Kore et al. 2024 ) . Preliminary efforts to leverage self-supervised learning (SSL) to improve fairness in multimodal use cases have proven promising ( Yfantidou et al. 2024 ) . However, they have mainly experimented with tabular data as opposed to attempting to integrate data from different modalities , length and temporality . Given that most datasets are only releasing the extracted features due to privacy concerns, researchers will have no control over how the features are extracted or processed which makes addressing the afore-mentioned gaps timely, pressing and relevant. In addition, the literature has not explored how SSL can be used in a multimodal setting where modalities are entirely different in terms of content and length (e.g., electroencephalogram (EEG) and audio in MODMA) (Gap 2) . The only study on SSL for multimodal fairness ( Yfantidou et al. 2024 ) applied SSL on tabular data, treating different modalities as if they are unimodal data. However, with entirely different modalities such as EEG and audio data, processing needs to be tailored to the nature and length of the signals of each modality. Contributions. In this work, we address the aforementioned challenges by introducing a novel subject-aware self-supervised method that mitigates bias in multimodal settings (Fig. 1 ). We base our approach on the Variance-Invariance-Covariance Regularization (VICReg) loss ( Bardes, Ponce, and LeCun 2022 ) which has been shown to be effective in learning useful representations in multimodal settings. We further exploit a key underpinning currently missed within the literature: different modalities may contain varying levels of individual and sensitive-attribute dependent information which can guide fairer and more robust learnt representations at different time-points . We do so in two ways. First, to improve fairness , we perform subject-aware changes on the loss function such that the variance term reduces its reliance on the protected attribute as a trivial solution, the invariance term ensures consistent predictions for similar individuals, and the covariance term minimizes correlational dependence on the protected attribute. Second, to address data heterogeneity in multimodal healthcare data of different modality, length and temporality, we make architectural changes, via (i) segment-based encoding and (ii) segment-based pooling such thatwe are still able to learn good and fair representations from modalities of variable feature length with temporality varying levels of task-relevant signals which are prevalent in healthcare settings. Our key contributions are as follows: • We are the first to investigate SSL-based methods for multimodal fairness in healthcare using non-tabular, variable-length data (e.g. EEG and audio). • We propose FairWell , a novel subject-aware method that extends VICReg ( Bardes, Ponce, and LeCun 2022 ) for ML fairness and addresses Gap 1 and Gap 2. • We demonstrate that our proposed modality agnostic method not only works on a variety of heterogeneous data (EEG, audio, visual, tabular etc.), but is also robust to ML in healthcare challenges (e.g. data drift, domain shift and data scarcity) and provides tangible improvements to the performance-fairness Pareto frontier . Approach Evaluation Fairness Measures [.4pt/2pt] Study Task MM Modality SSL BM VL AU-ROC SP EOpp EOdd EAcc Alasadi et al. ( Alasadi et al. 2020 ) Cyberbullying Detection ✓ VT ✓ ✓ ✓ ✓ Schmitz et al. ( Schmitz, Ahmed, and Cao 2022 ) Emotion Detection ✓ AVT ✓ ✓ ✓ Yan et al. ( Yan, Huang, and Soleymani 2020 ) Personality Assessment ✓ AV ✓ ✓ ✓ ✓ Kathan et al. ( Kathan et al. 2022 ) Humour Recognition ✓ AV ✓ ✓ ✓ Chen et al. ( Chen et al. 2023 ) Recommendation ✓ AVT ✓ ✓ ✓ ✓ Janghorbani et al. ( Janghorbani and De Melo 2023 ) Vision-Language Models ✓ VT ✓ Pena et al. ( Peña et al. 2023 ) Automatic Recruitment ✓ VT ✓ ✓ ✓ [.4pt/2pt] Barker et al. ( Barker, Bethell, and Kazakov 2024 ) Tabular & Language tabular, T ✓ ✓ ✓ ✓ ✓ ✓ Yfantidou et al. ( Yfantidou et al. 2024 ) Human-centred datasets tabular ✓ ✓ FairWell Healthcare ✓ AV, A-EEG, tabular ✓ ✓ ✓ ✓ ✓ ✓ ✓ ✓ Table 1: Comparative Summary with existing Multimodal Fairness SSL studies. Abbreviations (sorted): A: Audio. BM: Bias Mitigation. EAcc: Equal Accuracy. EEG: Electroencephalogram. EOdd: Equalised Odds. EOpp: Equality