Reliability-Aware Weighted Multi-Scale Spatio-Temporal Maps for Heart Rate Monitoring

Arpan Bairagi, Rakesh Dey, Siladittya Manna, Umapada Pal · 2026

Read Reliability-Aware Weighted Multi-Scale Spatio-Temporal Maps for Heart Rate Monitoring by Arpan Bairagi, Rakesh Dey, Siladittya Manna, Umapada Pal (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-26836.txt.

Introduction

Bairagi and co-authors aim to measure heart rate from face video without any contact sensor. Such signals break down under motion, shadows, and changing light. Their weighted map rates each pixel's reliability and suppresses noisy regions. A contrastive learning setup then trains a Swin-Unet on these maps. A wavelet map that keeps motion but removes pulse cues serves as the negative example. On public benchmarks, the method cut heart rate error below prior self-supervised approaches.

Catalog record

CREATORArpan Bairagi, Rakesh Dey, Siladittya Manna, Umapada Pal
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE25437
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.26836

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Reliability-Aware Weighted Multi-Scale Spatio-Temporal Maps for Heart Rate Monitoring Arpan Bairagi, Rakesh Dey, Siladittya Manna, Umapada Pal Abstract Remote photoplethysmography (rPPG) allows for the contactless estimation of physiological signals from facial videos by analyzing subtle skin color changes. However, rPPG signals are extremely susceptible to illumination changes, motion, shadows, and specular reflections, resulting in low-quality signals in unconstrained environments. To overcome these issues, we present a Reliability-Aware Weighted Multi-Scale Spatio-Temporal (WMST) map that models pixel reliability through the suppression of environmental noises. These noises are modeled using different weighting strategies to focus on more physiologically valid areas. Leveraging the WMST map, we develop an SSL contrastive learning approach based on Swin-Unet, where positive pairs are generated from conventional rPPG signals and temporally expanded WMST maps. Moreover, we introduce a new High-High-High (HHH) wavelet map as a negative example that maintains motion and structural details while filtering out physiological information. Here, our aim is to estimate heart rate (HR), and the experiments on public rPPG benchmarks show that our approach enhances motion and illumination robustness with lower HR estimation error and higher Pearson correlation than existing Self-Supervised Learning (SSL) based rPPG methods. † † address: ⋆ CVPR Unit, Indian Statistical Institute, Kolkata, India † Department of Computational and Data Sciences, Indian Institute of Science, Bengaluru, India 1 Introduction Remote PPG deals with non-contact estimation of different kinds of physiological signals, such as heart rate and SpO2 (peripheral capillary oxygen saturation), which can be analyzed from small colour variations on facial skin due to the blood volume pulse (BVP). “rPPG" is contactless and captures small colour variation, but provides a poor Signal to Noise Ratio (SNR) when affected by different environmental noises. Traditional signal-processing-based and region-based approaches [ 18 , 10 , 3 , 4 , 24 , 17 ] rely on handcrafted features, but do not perform well under unconstrained situations. Whereas, recent deep learning methods [ 16 , 12 , 5 ] provide robust solutions to obtain better rPPG signal by learning on direct videos or preprocessed spatio-temporal embeddings. Still, these deep learning methods rely on large-scale annotated data, and acquisition of such large-scale data is challenging, which led us to the use of SSL methods [ 6 , 25 , 22 , 11 , 20 , 21 ] . SSL-based rPPG methods mainly concentrate on learning invariances based on contrastive learning through extensive data augmentations, while also using conventional pre-processing techniques such as Multi-Scale Spatio-Temporal (MST) map [ 16 ] , which do not model noises properly. Moreover, the current SSL-based rPPG approaches utilize generic negative augmentations that are either trivially separable or lack the capability to maintain realistic facial dynamics. To resolve the challenges mentioned above, our contributions are: • We have proposed an advanced pre-processing technique for weighting pixels based on their reliability for important physiological signals, the Weighted Multi-Scale Temporal (WMST) Map, which produces better SNR. To prove that our proposed map provides better SNR, we have conducted several experiments, and the results show that our proposed technique outperforms the existing MST map for different statistical testing. • We have used a high-frequency wavelet-based negative, High-High-High (HHH) wavelet map for contrastive learning. Such HHH wavelet map models motion and noises, which are unimportant for rPPG. To our knowledge, such an HHH map has not been used earlier. 2 RELATED WORKS Initial deep learning-based rPPG methods used to handle unconstrained situations far better than traditional methods, leading towards better performances. However, these deep learning-based models heavily relied on ground-truth physiological annotations to produce more feasible outcomes. To overcome this problem, recent works have investigated self-supervised learning (SSL) methods, which discourage the use of labeled data. Gideon et al. [ 6 ] were among the first few who proposed a contrastive learning methodology to predict meaningful physiological representations in the direct form of unlabeled facial videos. Following [ 6 ] Yue et al. [ 25 ] extended this concept and created a self-supervised method that uses spatial and temporal consistency across various regions of the face to enhance robustness for physilogical enhancement. Speth et al. [ 22 ] , on the other hand, chose to completely steer clear of the contrastive learning framework and instead opted for a non-contrastive framework, where the model learns to enforce consistency between different views of the same video without the need to explicitly construct negative samples. Li et al. [ 11 ] proposed using traditional signals for helping the model to physiology-aware weights. Later, RS-rPPG [ 20 ] and [ 21 ] extended the idea proposed by [ 11 ] by using different physiology-based traditional positives and negatives augmentations, leading to improved robustness in rPPG estimation. In contrast, our work improves robustness by enhancing spatiotemporal signal representations through WMST map construction and wavelet-aware augmentation for a contrastive learning based self-supervised learning framework without relying on ground truth (GT) labels. 3 PROPOSED METHODOLOGY Our proposed framework consists of two key components: (1) addition of WMST map , which models different unconstrained situations harmful for rPPG estimation, and (2) for contrastive learning, use of HHH wavelet map that retains motion and structural patterns. 3.1 Reliability-Aware Weighted MST Map Generation Remote PPG can be modelled by very small colour changes, making the signal highly sensitive to environmental noise. In order to handle these issues, we propose an improvement of the MST map [ 16 ] , the WMST map. Following the idea of the MST map, each frame ( I ) of the video is converted into YUV and stacked with RGB, as YUV tends to work better for physiological signals [ 15 ] . For each frame, we extract six regions of interest (ROIs): forehead, mouth, left cheek (two regions), and right cheek (two regions). A binary mask is created for each of the ROIs. From this set of ROIs, all possible non-empty subsets are created, and from each of these subsets, one value is pooled, resulting in better physiological signals [ 16 ] . In rPPG, the reliability of a pixel depends on environmental noises such as reflection, sensor noise, shadow, and also the likelihood that the pixel is from the skin region. Unlike the MST map that uses mean pooling of all of the pixels within an ROI, we have modelled the reliability of a pixel using different well-established statistical and mathematical approaches for pooling. We estimate the skin probability of each pixel, and we adopt a chrominance-based skin modelling [ 7 ] . From the U and V channel of the ROIs, a chrominance mean vector \boldsymbol{\mu}=[\mu_{U},\mu_{V}]^{\top} and a covariance matrix \boldsymbol{\Sigma} is computed from Z=[U,V]^{\top} . Then the Mahalanobis distance is used to capture the skin weight w_{\mathrm{skin}}=\exp\!\left(-(\mathbf{z}-\boldsymbol{\mu})^{\top}\boldsymbol{\Sigma}^{-1}(\mathbf{z}-\boldsymbol{\mu})\right),\quad\mathbf{z}\in Z. (1) Pixels with strong edge responses signify non-physiological textures or motion boundaries. We employ the Stationary Wavelet Transformation (SWT) to suppress noise and motion artifacts [ 9 ] . Using SWT, we obtain the (HL,LH,HH) subband of the luminance channel. The local edge energy at each pixel is then suppressed by w_{\mathrm{edge}}=\exp(-\sqrt{HL^{2}+LH^{2}+HH^{2}}). (2) For identifying reflection and shadow HSV colour space is used, as bright and weakly saturated pixels are exhibited by specular highlights: s_{\mathrm{spec}}=(1-S)\cdot V, (3) where, S and V are the channels of HSV. Pixels affected by specular reflections

Read the complete text