Wearable Sleep-Stage Classification: A Systematic Analysis of Signal Modalities, Representations, and Multimodal Fusion Authors: Unaza Tallal, Rupesh Agrawal, Shruti Kshirsagar Published: 2026 ABSTRACT Accurate sleep staging is essential for characterizing sleep architecture, yet conventional polysomnography limits scalable and long-term sleep monitoring. Wearable sensors offer a practical alternative, but their performance depends on the physiological modalities used and how the signals are represented. This study investigates the effects of signal modality, feature representation, and sleep-stage granularity on subject-independent wearable sleep staging. Using DREAMT, we compare handcrafted features, short-time Fourier transform (STFT) representations, and their fusion across three-, four-, and five-class tasks. Handcrafted features were modeled using a CNN–BiGRU framework, while STFT representations of BVP and accelerometry were learned using a deep time–frequency model. All DREAMT experiments were evaluated using five-fold subject-independent cross-validation, with Macro-F1 and Cohen’s κ emphasized to account for class imbalance. We further evaluated optical pulse signal-based classification using the larger MESA Sleep cohort. STFT improved class-balanced performance over handcrafted features, while BVP was the strongest individual STFT modality, particularly for REM. For the three-, four-, and five-class tasks, handcrafted BVP + ACC achieved Macro-F1 scores of 0.5057, 0.4005, and 0.3366, respectively, compared with 0.6404, 0.5083, and 0.4376 using STFT. Hybrid fusion achieved Macro-F1 scores of 0.6476, 0.5108, and 0.4464, respectively. More complex fusion strategies did not improve Macro-F1 over simple concatenation. In MESA, increasing the cohort from 100 participants to the full eligible cohort improved pulse-based sleep-stage classification, although N1 and N3 remained comparatively difficult. Overall, the results show that effective signal representation and complementary physiological information are more important than simply increasing the number of modalities or model complexity. 1. Introduction Sleep staging provides important information about sleep architecture and is widely used to assess sleep quality and sleep-related disorders. Standard sleep staging divides a recording into Wake, rapid eye movement (REM), and non-REM stages N1, N2, and N3 according to established scoring criteria [1]. Polysomnography (PSG) remains the clinical reference standard because it combines electroencephalography, electrooculography, electromyography, and other physiological measurements. However, PSG requires specialized equipment and trained personnel, which limits its practicality for repeated and long-term sleep monitoring. Wearable devices provide a less intrusive alternative for continuous sleep monitoring. Modern wrist-worn devices can record blood volume pulse (BVP), accelerometry (ACC), electrodermal activity (EDA), and skin temperature, as well as cardiac measures such as heart rate (HR) and interbeat intervals (IBI) [2]. These modalities capture different aspects of sleep physiology. Cardiac signals reflect autonomic changes, while accelerometry primarily captures movement. EDA and skin temperature provide additional information related to sympathetic and thermoregulatory activity [3]. Recent wearable studies increasingly combine PPG and accelerometry for multi-stage sleep classification, although performance varies considerably across sensor configurations and evaluation settings [4]. A key challenge is how to represent wearable signals. Traditional approaches use handcrafted features to characterize pulse morphology, cardiac variability, movement, electrodermal activity, and temperature. Deep learning approaches can instead learn representations from raw signals or time–frequency transformations. Short-time Fourier transform (STFT) representations preserve changes in spectral content over time and provide structured inputs for deep models. However, it remains unclear whether STFT representations consistently provide an advantage over handcrafted physiological features and whether combining the two representations captures complementary information. The contribution of each modality may also depend on sleep-stage granularity. A signal that effectively separates Wake, NREM, and REM may provide less information when NREM is divided into N1, N2, and N3. Similarly, adding more modalities or using more complex fusion strategies may introduce redundant information without improving class-balanced performance. Performance may further depend on cohort size and signal-acquisition conditions. These factors make it difficult to determine whether improvements arise from the physiological modality, signal representation, fusion strategy, or available training data. In this study, we systematically investigate how physiological modality, signal representation, and sleep-stage granularity affect subject-independent wearable sleep staging. We use DREAMT for multimodal experiments with wrist-recorded BVP, ACC, EDA, skin temperature, HR, and IBI [2]. We compare handcrafted and STFT-based representations, quantify the contribution of individual and combined modalities, and evaluate whether their fusion provides complementary information. We conduct these analyses across three-, four-, and five-class sleep-stage tasks. We further evaluate PPG-based sleep staging using the MESA Sleep cohort to examine performance under a different pulse-signal acquisition setting and with a substantially larger cohort [5,6]. The main contributions of this study are as follows:We systematically compare handcrafted and STFT-based wearable signal representations across three-, four-, and five-class sleep-stage classification tasks.We quantify the contribution of individual and combined wearable modalities and examine how their contributions change with sleep-stage granularity.We evaluate whether handcrafted physiological features provide complementary information to learned STFT representations and compare alternative fusion strategies.We extend the pulse-based analysis to the larger MESA Sleep cohort to examine performance under a different PPG acquisition setting and training scale. We systematically compare handcrafted and STFT-based wearable signal representations across three-, four-, and five-class sleep-stage classification tasks. We systematically compare handcrafted and STFT-based wearable signal representations across three-, four-, and five-class sleep-stage classification tasks. We quantify the contribution of individual and combined wearable modalities and examine how their contributions change with sleep-stage granularity. We quantify the contribution of individual and combined wearable modalities and examine how their contributions change with sleep-stage granularity. We evaluate whether handcrafted physiological features provide complementary information to learned STFT representations and compare alternative fusion strategies. We evaluate whether handcrafted physiological features provide complementary information to learned STFT representations and compare alternative fusion strategies. We extend the pulse-based analysis to the larger MESA Sleep cohort to examine performance under a different PPG acquisition setting and training scale. We extend the pulse-based analysis to the larger MESA Sleep cohort to examine performance under a different PPG acquisition setting and training scale. The paper is structured as follows. Section 2 reviews the related work on wearable sleep staging and signal representations. Section 3 describes the methodology, including the datasets, signal preprocessing, feature representations, and model architectures. Section 4 presents and discusses the experimental results. Finally, Section 5 summarizes the main findings and concludes the paper. 2. Related Work 2.1. Wearable Methods for Sleep Monitoring Early wearable sleep-monitoring methods relied mainly on wrist actigraphy [7]. Actigraphy uses accelerometer-derived movement to estimate sleep and wake. It performs reasonably well for binary sleep and wake classification [8]. However, movement alone provides limited information for distinguishing REM from NREM sleep. Sundararajan et al. [9] used random forests with wrist accelerometry from more than 130 participants. Their results showed that multi-stage classification from movement alone remained challenging. Palotti et al. [10] reached a similar conclusion in a large, multi-cohort benchmark of sleep-wake classifiers, where machine-learning models trained on actigraphy performed well for wake detection but offered little multi-stage resolution. The addition of photoplethysmography (PPG) introduced cardiac information into wearable sleep staging [11]. PPG-derived heart rate and pulse-rate variability capture autonomic changes across sleep stages, reflecting differences in sympathetic and parasympathetic activity between NREM and REM sleep [11]. Walch et al. [12] combined PPG-derived heart rate with wrist accelerometry. The combined signals improved sleep-stage classification compared with accelerometry alone. Olsen et al. [13] later proposed a flexible deep learning architecture for temporal sleep-stage classification using accelerometry and PPG together, and reported consistent gains from combining the two signals across recording conditions. Motin et al. [14] likewise showed that multi-stage sleep classification was feasible from a PPG sensor alone, without accompanying accelerometry, though performance remained below that of multi-modal approaches. Sano et al. [15] also combined actigraphy, cardiac, and electrodermal activity (EDA) signals. Their results showed that multimodal physiological information improved sleep-stage classification. Boe et al. [16] reported a related finding using a wireless, multi-sensor wearable system, where combining physiological streams improved automated staging over single-sensor baselines. Later studies also reported complementary information from PPG and accelerometry, although performance varied across sleep stages [17]. A recent review confirmed the growing use of PPG and accelerometry for wearable sleep staging [4]. It also highlighted the limited ability of accelerometry alone to distinguish multiple sleep stages. However, large differences remain in sensor configurations, algorithms, and evaluation protocols across studies. These differences make direct comparisons difficult. Data scarcity compounds this problem, since PPG-labeled sleep data remain far less common than ECG data. Radha et al. [18] and Li et al. [19] both addressed this gap through transfer learning, pretraining on ECG-derived heart-rate information before fine-tuning on PPG, and reported improved staging accuracy relative to training on PPG alone. Signal quality is a further practical constraint: Orphanidou et al. [20] proposed signal-quality indices for PPG and ECG that are now commonly used to exclude unreliable wearable recordings before staging. EDA and skin temperature have received less attention. EDA shows stage-dependent changes during sleep, including characteristic activity during NREM sleep [3]. Herlan et al. [21] confirmed this pattern directly, showing that EDA activity differed systematically between sleep and wake and offered some value for sleep/wake classification specifically [21]. Skin temperature also changes with thermoregulatory activity across the sleep cycle. These signals may therefore provide information beyond cardiac and movement signals. More broadly, existing multimodal wearable studies often evaluate a fixed combination of sensors, whereas fewer systematically compare the incremental contributions of individual and combined modalities within the same evaluation framework. 2.2. Feature Representation and Deep Learning Sleep-staging approaches broadly split along how much of the representation is hand-specified versus learned, though in practice the two ends of this spectrum share more structure than the split suggests. The classical paradigm treats staging as classification over hand-engineered features from 30 s epochs. For cardiac signals, this usually means time- and frequency-domain HRV or PRV statistics: mean and SD of interbeat intervals, RMSSD, low/high-frequency power ratios [22]. This feature set reflects a well-established autonomic pattern: HRV power shifts characteristically between REM and deep NREM sleep [22]. For accelerometry, this comprises activity counts, signal magnitude area, and axis-wise variance, borrowed from older actigraphy scoring [9]. EDA features come from skin-conductance level and the rate, amplitude, and duration of skin-conductance responses, following Sano et al.’s characterization of EDA during sleep [3]. Temperature features are simpler still, mostly level and slope, since thermoregulation moves slowly compared to cardiac or electrodermal signals. Modulation-spectrogram features are an alternative to raw signal or STFT, capturing slower amplitude-modulation patterns on top of a signal’s short-term spectrum [23]; Tallal et al. applied this to single-channel EEG from DREAMT and found it held up much better than STFT and wavelet baselines once participants were split by apnea severity [24], matching earlier work on modulation-spectrum robustness in other physiological and wearable tasks [25,26], and motivating modulation features as a next step for our BVP/ACC pipeline. Once extracted, these features are typically classified with either non-sequential models like random forests or sequence models that exploit predictable sleep-stage transitions. Topalidis et al. [27] used a random forest on HRV features derived from low-cost wearable ECG and PPG sensors, and additionally applied a random forest to flag poor-quality interbeat intervals before staging, showing that feature quality control matters as much as the classifier itself. Radha et al. [28] instead applied an LSTM directly to HRV features across a large, diverse dataset, and found that modeling long-term cardiac sleep architecture outperformed non-temporal and short-term recurrent classifiers. A consistent finding across this literature is that cardiac features (HR, PRV) carry the most useful information for REM and wake; accelerometry mainly helps with wake through movement [12], and EDA/temperature give smaller, less reliable gains that depend on what else is already in the feature set [21]. Deep learning approaches skip this handcrafted step and learn representations directly from raw or minimally processed physiological signal [29,30]. However, they inherit the same underlying design problem: how to capture local-epoch structure and inter-epoch transitions. They typically address this problem using related architectural building blocks (convolution for local shape, recurrence or attention for sequence) [31,32]. DeepSleepNet paired CNNs for shape-invariant EEG features with bidirectional LSTMs for epoch transitions, and matched or beat hand-engineered pipelines across datasets [33]. EEGNet took a more general route: a compact CNN using depthwise and separable convolutions, originally built for BCI tasks but now widely reused as a lightweight EEG backbone, including for sleep staging [34]. Eldele et al. proposed AttnSleep, combining a multi-resolution CNN with a multi-head attention encoder, and outperformed prior single-channel EEG models on three public datasets [35]. U-Time treated staging as dense segmentation with a U-Net-style encoder–decoder, and was more robust to hyperparameter changes than recurrent alternatives across seven PSG datasets [36]. SleepPPG-Net brought this end-to-end approach to wearables, running a residual plus temporal CNN directly on raw PPG for solid four-class staging [37]. Habib et al. also trained a CNN on raw PPG, achieving competitive accuracy without morphology features [38]. More recently, SleepMFormer replaced full self-attention with a task-driven, sparse Transformer encoder plus contrastive pretraining, cutting training and inference time by up to a third with no loss in accuracy on Sleep-EDF, PhysioNet, and SHHS [39]. NanoSleep takes a different angle: a Sinc-convolutional front end plus a gated dilated temporal CNN and CRF decoding, matching CNN–BiLSTM or transformer accuracy on single-channel EEG with far fewer parameters, useful for on-device staging [40]. These results shaped two choices here. First, we use an STFT representation of BVP and accelerometry magnitude with a U-Net-style spectral encoder, following U-Time [36] but more closely aligned with SleepPPG-Net’s wearable setting [37]. Second, for handcrafted features, we use a CNN–BiGRU, in DeepSleepNet’s spirit [33], to capture local patterns and inter-epoch transitions together. We then fuse the two branches, deep spectral and handcrafted recurrent, to test whether learned and physiologically grounded features add anything beyond either alone. Learned EEG/PPG representations [24,33,34,35,36,37,38,39,40] and handcrafted multimodal pipelines [12,15,17] have each been validated on their own datasets. 2.3. Generalization A separate and increasingly emphasized concern is whether a trained model holds up outside its training distribution, across subjects, sensors, and clinical populations, rather than just on a held-out split of the same cohort. Some of this variability is addressable by adapting to a subgroup rather than training one model for everyone: Hossain and Kshirsagar fine-tuned a convolutional-recurrent model on demographic subgroups (age, sex, OSA severity) within a single 100-subject cohort, and found 35 of 37 subgroup-tuned models outperformed the pooled baseline, pointing to within-cohort population structure, not just architecture, as a real driver of accuracy [41]. Moreover, Refs. [24,42] showed apnea-based cohort classification for five-stage sleep classification. Van Der Aar et al. tested the complementary, cross-cohort question directly, comparing pretraining, training-from-scratch, and fine-tuning for single-channel EEG staging across eight target datasets; transfer learning gave little advantage over training from scratch in most age-matched cohorts, and only became clearly necessary in the smallest, suggesting that channel and population mismatch mainly hurt when combined with limited target data rather than independently [43]. Other work treats domain shift as a modeling problem to solve explicitly rather than data to fine-tune on. STDA-Net combines spectrogram representations with adversarial domain alignment to improve cross-dataset EEG staging without target-domain labels [44], and SleepDG aligns features at both the epoch and sequence level across multiple source datasets, reporting improved performance on domains never seen during training [45]. SleepPPG-Net2 applied the same multi-source idea to wearable PPG specifically, training across six datasets (2574 recordings) and improving cross-dataset generalization (Cohen’s kappa) by up to 21% compared with single-source training approaches [46]. Despite these advances, there is still limited work that examines wearable sleep staging from all of these perspectives within the same study. Most existing studies focus on a particular sensing modality, representation method, model architecture, or fixed multimodal configuration. In contrast, the present study systematically compares handcrafted and STFT-based representations, evaluates different combinations of wearable signals and hybrid fusion strategies, and investigates three-, four-, and five-class sleep-stage classification within a common subject-independent framework. This allows us to examine not only which approaches perform best, but also whether adding more physiological modalities or using more complex models consistently yields better-balanced sleep-stage classification. 3. Methods and Materials In this section, we describe the datasets, signal preprocessing, feature representations, model architectures, training procedure, and evaluation methods used in this study. The overall study design, including the DREAMT-based multimodal analysis and the independent MESA comparative analysis, is summarized in Figure 1. 3.1. Dataset We used two sleep datasets: DREAMT [47], available through PhysioNet, and the Multi-Ethnic Study of Atherosclerosis (MESA) Sleep Ancillary Study, accessed through the National Sleep Research Resource (NSRR) [5,6]. DREAMT was used for the primary multimodal, feature-representation, and fusion experiments, while MESA was used to evaluate pulse-based sleep-stage classification in a larger cohort and under a different acquisition setting. Participant demographics and sleep characteristics are summarized in Table 1. 3.1.1. DREAMT Dataset DREAMT (Dataset for Real-time Sleep Stage Estimation using Multisensor Wearable Technology) [47] is a publicly available dataset containing synchronized wrist-worn physiological signals from 100 adults undergoing overnight clinical sleep assessment. Signals were recorded using the Empatica E4 wristband and include blood volume pulse (BVP), triaxial accelerometry (ACC), electrodermal activity (EDA), skin temperature, heart rate (HR), and interbeat interval (IBI), together with PSG-derived sleep-stage annotations assigned by expert sleep technicians. The multimodal recordings were used to investigate the individual and combined contributions of wearable modalities and to compare handcrafted, STFT-based, and hybrid feature representations. After applying the signal-quality criteria, 99 participants were retained for analysis. 3.1.2. MESA Sleep Dataset The MESA Sleep Ancillary Study was accessed through the NSRR following approval of the required Data Access and Use Agreement [5,6]. The dataset contains overnight in-home polysomnography recordings from 2056 participants. Although multiple physiological signals are available, only the photoplethysmographic (PPG) waveform was used in this study to provide a pulse-based evaluation complementary to the wrist BVP available in DREAMT. The PPG signal is provided as the Pleth channel and was recorded from the finger using a pulse-oximetry sensor at 256 Hz. After applying the inclusion and signal-quality criteria, 2002 participants were retained for analysis. Two MESA cohorts were evaluated: a 100-participant subset (MESA-100) and the complete eligible cohort (MESA-all; n=2002). MESA-100 provided a cohort size comparable to DREAMT, while MESA-all enabled evaluation using a substantially larger training population. The construction of MESA-100 is described in Section 3.3. MESA was analyzed independently as an additional cohort rather than as a direct cross-dataset external validation of models trained on DREAMT. 3.1.3. Participant Demographics and Sleep Parameters Participant demographic and sleep characteristics for DREAMT, MESA-100, and MESA-all are summarized in Table 1. In addition to these cohort differences, DREAMT and MESA also differ in data-acquisition conditions, including pulse-signal sensor placement, sampling configuration, and the set of available wearable modalities. These acquisition differences may also contribute to the observed performance differences between the datasets. 3.2. Sleep-Stage Definitions and Label Harmonization Both datasets provide sleep-stage labels for consecutive 30 s epochs. The labels include Wake, N1, N2, N3, and REM. We excluded epochs labeled as movement, preparation, unknown, or unscored. We evaluated three sleep-stage classification tasks. Table 2 summarizes the label mappings. For the five-class task, we retained the original five stages. For the four-class task, we combined N1 and N2 as Light sleep and retained N3 as Deep sleep. For the three-class task, we combined N1, N2, and N3 as NREM sleep. We applied the same label mappings to both datasets. 3.3. Data Quality Control and Cohort Selection We screened both datasets for invalid signals, missing sleep-stage labels, recording discontinuities, and alignment with the 30 s scoring grid. We excluded invalid epochs and processed continuous recording segments separately to avoid crossing signal gaps. We did not impute missing sleep-stage labels. For DREAMT, retained epochs were required to contain a complete 30 s interval with a single valid sleep-stage label. Signal-integrity checks verified finite BVP and ACC values, timestamp continuity, and valid ACC sampling structure. One participant was excluded because of accelerometer failure, leaving 99 participants, 102 continuous sequences, and 79,468 valid epochs. Table 3 summarizes the resulting class distribution, showing that the five-class distribution is imbalanced, with N3 being the least represented stage and N1 also occurring less frequently than Wake and N2. We considered this class distribution when selecting balanced evaluation metrics for the study. For MESA, PPG recordings were audited for signal and annotation quality. Recordings were flagged for EDF–annotation duration mismatches greater than 1 s, non-finite samples, more than 50% identical adjacent samples, flatlines longer than 10 s, PPG interquartile range below 10−4 of the physical signal range, saturation affecting more than 5% of samples, or pulse-band power ratio below 0.05. Audit flags were not automatic subject-level exclusion criteria; we retained usable portions as independent continuous sequences. Participants were included when at least 780 usable epochs (6.5 h) were available and at least one continuous sequence contained 360 epochs (3 h). After quality control, 2002 participants remained. We constructed MESA-100 by selecting 25 participants from each AHI category (normal, mild, moderate, and severe) to provide a cohort size comparable to DREAMT, and balancing the subset by sex. We used a fixed random seed of 2022 for reproducible sampling. 3.4. Signal Preprocessing We processed each continuous recording sequence independently and aligned the signals with the 30 s sleep-stage annotations. DREAMT used separate preprocessing pipelines for the STFT and handcrafted representations, while MESA used PPG for STFT-based classification. 3.4.1. DREAMT Signal Preprocessing For the STFT representation [48,49], we processed BVP and ACC following [13]. We low-pass filtered BVP at 12 Hz using a fourth-order zero-phase Chebyshev Type-I filter and downsampled it from 64 to 32 Hz. We then applied a fifth-order zero-phase Butterworth band-pass filter between 0.1 and 8 Hz. Finally, we applied a centered 300 s adaptive IQR normalization and clipped the normalized signal to [−20,20]. We restored ACC to its native 32 Hz sampling rate and normalized each axis using participant-specific IQR statistics. The processed BVP and ACC signals were then used to generate the STFT representations. For handcrafted features, we retained BVP at 64 Hz and applied a fourth-order zero-phase Butterworth band-pass filter between 0.5 and 8 Hz before pulse detection. We restored ACC, EDA, skin temperature, and HR to their native sampling rates. We decomposed EDA into tonic and phasic components using a 0.05 Hz low-pass filter. We derived interbeat intervals from consecutive BVP peaks and retained intervals between 0.30 and 2.00 s for PRV analysis. All processing remained within continuous recording sequences. 3.4.2. MESA Signal Preprocessing We processed the MESA finger PPG using the same pulse-signal pipeline used for the DREAMT STFT representation. We low-pass filtered the 256 Hz PPG at 12 Hz, downsampled it to 32 Hz, and applied a 0.1–8 Hz Butterworth band-pass filter. We then applied the centered 300 s adaptive IQR normalization and clipped the signal to [−20,20]. We used the resulting PPG signal to generate the STFT. 3.5. Feature Representation and Extraction We generated two complementary representations: STFT-based time–frequency representations and handcrafted physiological features. We aligned both representations with the corresponding 30 s sleep-stage epochs and generated them independently within each continuous recording sequence. 3.5.1. STFT-Based Time–Frequency Representation Following [13], to maintain methodological consistency with an established wearable sleep-staging framework, we computed the STFT continuously within each valid recording sequence rather than independently for each 30 s epoch. We used a 10 s Blackman window with 9 s overlap, producing one spectral frame per second. Each sleep-stage epoch therefore corresponded to 30 consecutive spectral frames. For BVP and PPG, we used a 512-point Fourier transform and retained 64 frequency bins below 4 Hz. We converted spectral power to the natural logarithmic scale. We applied the same STFT parameters to ACC. For ACC, we evaluated three representations: axis-averaged, magnitude, and separate triaxial spectrograms. For the magnitude representation, we computed(1)amag(t)=ax(t)2+ay(t)2+az(t)2.For the axis-averaged representation, we averaged the three axis-specific spectrograms element-wise. For the triaxial representation, we retained the three spectrograms as separate input channels. During model training, we used sequences of up to 1024 consecutive epochs (approximately 8.5 h). We zero-padded shorter sequences and masked the padded positions during training and evaluation. To illustrate the resulting time–frequency representations, Figure 2 shows a representative DREAMT segment containing the raw BVP and ACC signals, their corresponding STFT representations, and the aligned five-class sleep-stage sequence. The example demonstrates how changes in the physiological signals and their spectral content correspond temporally with transitions between sleep stages. 3.5.2. Handcrafted-Feature Extraction We extracted 112 handcrafted features from DREAMT to characterize cardiac, movement, electrodermal, and skin-temperature activity. Most features were computed within each 30 s epoch. PRV and selected temporal features used neighboring intervals within the same continuous recording sequence and did not cross masked epochs or recording gaps. The feature bank contained 37 cardiac, 41 ACC, 20 EDA, and 14 skin-temperature features (Table 4). Cardiac features described BVP morphology, interbeat intervals, HR, and PRV [50]. ACC features characterized movement intensity, orientation, temporal changes, and frequency content [51]. EDA features captured tonic and phasic activity [52], while temperature features described level, variability, trend, and temporal changes [8]. The complete feature definitions are provided in the Abbreviations. 3.5.3. Feature Assembly and Preprocessing We aligned modality-specific features by participant, sequence, and epoch and combined them into a 112-dimensional feature vector. Pairwise Spearman correlation identified 59 feature pairs with |ρ|≥0.95, all within individual modalities. We retained all features because the analysis showed no highly correlated cross-modality pairs and the overall dimensionality remained moderate. We fitted missing-value imputation and feature scaling using only the training participants within each cross-validation fold. We then applied the fitted transformations unchanged to the validation and test participants. 3.6. Model Architectures This section describes the model architectures evaluated in this study, including the STFT-based U-Net, the handcrafted-feature model, and the proposed fusion model. 3.6.1. STFT-Based U-Net Model The STFT-based sleep-stage classifier was implemented following the U-Net-inspired architecture proposed by [13]. The baseline configuration used two modalities: BVP and ACC. We combined the spectrograms of these two modalities into two input channels and processed them together with a temporal encoder–decoder network. The input modalities contained 1024 consecutive 30 s epochs, with each represented by 30 one-second spectral frames and 64 frequency bins. The encoder gradually reduced the temporal and frequency dimensions while increasing the number of effectively learned feature maps. The decoder used skip connections to restore temporal resolution and combine features from the corresponding encoder level. The implementation used 10 encoder–decoder levels, 16 initial filters, a convolutional kernel of size 16×3, and a temporal frequency reduction factor of 2×2. This model served as both the STFT-based baseline and the spectral feature extractor for the proposed fusion framework. 3.6.2. Handcrafted-Feature Temporal Model To learn temporal patterns from handcrafted features, we used a temporal CNN–BiGRU model. We processed handcrafted features from consecutive valid 30 s epochs as continuous sequences from the same recording. Context windows were restricted to each participant’s continuous valid recording sequence and did not extend across recording gaps or masked epochs. To handle sequence boundaries, we padded the context using only values from the same valid sequence. To capture short-term patterns between neighboring epochs, the input feature sequence was first processed by a temporal convolutional encoder with 64 output channels. The generated representation was then passed through a bidirectional gated recurrent unit with 64 hidden units in each direction. The forward and backward hidden states were combined to form a 128-dimensional embedding for each target epoch. We then applied a dropout rate of 0.25 to reduce overfitting before passing the embedding to the final fully connected classification layer. 3.6.3. Proposed Dual-Branch STFT–Handcrafted Fusion Model The proposed fusion model integrates complementary information from STFT-based and handcrafted feature representations and assesses the contribution of multimodal fusion over single-modality configurations. The STFT branch learns time–frequency patterns directly from the signals and captures longer-term sleep-stage dependencies [13], whereas the handcrafted branch encodes physiologically motivated morphological, statistical, temporal, and frequency-domain characteristics. These two representations therefore provide complementary views of the wearable signals, with one learned from the time–frequency structure and the other guided by established physiological descriptors. Figure 3 shows the architecture. We trained the two branches separately and used the checkpoints that performed best on the validation set to generate an embedding for each valid 30 s epoch. We then aligned the STFT and handcrafted embeddings using the participant ID, recording sequence ID, and the epoch position within the sequence. We included an epoch in the fusion dataset only when these identifiers and the corresponding sleep-stage label matched across both branches. For epoch i, let ziSTFT denote the embedding generated by the STFT branch and ziHC denote the embedding generated by the handcrafted-feature branch. The aligned STFT and handcrafted embeddings were concatenated to form a joint feature representation [53]:(2)zi=ziSTFT∥ziHC, where ‖ denotes feature-wise concatenation.(3)z˜i=zi−μtrainσtrain, where μtrain and σtrain denote the feature-wise mean and standard deviation estimated from the training partition [54]. The validation and test sets used the same scaling parameters. During fusion training, the STFT and handcrafted encoders remained frozen, and only the multilayer perceptron (MLP) classifier was updated. The fusion MLP first applied layer normalization [55], followed by a fully connected layer with 128 units, GELU activation [56], and dropout. A second fully connected layer reduced the embedding to 64 dimensions, followed again by GELU activation and dropout. The final linear layer produced one logit for each sleep-stage class. The resulting logit vector can be expressed as(4)oi=fMLPz˜i, where fMLP(·) represents the fusion classifier and oi∈RC contains the logits for the C sleep-stage classes. During training, these logits were passed directly to the weighted cross-entropy loss. During inference, softmax was used to convert the logits into class probabilities [57]:(5)pi,c=exp(oi,c)∑k=1Cexp(oi,k), where pi,c is the predicted probability that epoch i belongs to class c. The final predicted sleep stage was then obtained as(6)y^i=argmaxcpi,c. 3.7. Model Training and Selection The training configuration of the individual branches and the proposed fusion model is summarized in Table 5. The dataset was split at the participant level to prevent data leakage, with all recordings and sequences from the same participant assigned to one partition. We reserved 15 participants as an independent holdout test set and used the remaining participants for five-fold cross-validation. In each fold, approximately 68–69% of the development participants were used for training, 12% for validation, and 19–20% for cross-validation testing. To improve reproducibility, all randomized procedures used a fixed seed of 2022. We estimated all preprocessing parameters only from the training partition. To reduce the effect of sleep-stage imbalance, we trained all models using effective-number-weighted cross-entropy loss with β=0.999. The validation set was used for model selection and early stopping, whereas the cross-validation test set was used for performance comparison. After the final model configuration was fixed, the holdout set was evaluated once to estimate performance on unseen participants. 3.8. Experimental Setup and Reproducibility We implemented all experiments in Python 3.9.21 using PyTorch 2.8.0 with CUDA 12.8. For model training, we used GPU resources of the ACCESS Delta high-performance computing system, with NVIDIA A40 GPUs (NVIDIA Corporation, Santa Clara, CA, USA) [58,59]. Additional preprocessing and experimental checks were performed on a local workstation equipped with an NVIDIA Quadro RTX 6000 GPU with 24 GB memory. Throughout the experiment, a fixed seed of 2022 was used, and the same participant-level data splits were maintained across the compared models to support reproducibility and fair comparison. 3.9. Evaluation Metrics We evaluated sleep-stage classification at the 30 s epoch level using accuracy, Macro-F1 [60], Cohen’s κ, and class-specific F1-score [61]. We also examined confusion matrices to identify common misclassifications between sleep stages. We used Macro-F1 as the primary model-selection metric because it gives equal weight to each class and is less affected by class imbalance than overall accuracy. However, Macro-F1 can decrease when performance on minority classes worsens, even if overall accuracy increases. Therefore, a higher accuracy together with a lower Macro-F1 may indicate greater emphasis on the more frequent classes at the expense of balanced performance across sleep stages. Accuracy measures the proportion of correctly classified epochs [62]. Macro-F1 is the unweighted mean of the class-specific F1-scores and was calculated as(7)Macro−F1=1C∑c=1CF1c, where C is the number of sleep-stage classes. Cohen’s κ measures agreement between predicted and reference labels while accounting for agreement expected by chance:(8)κ=po−pe1−pe, where po and pe denote the observed and expected agreement, respectively [63]. For cross-validation, we evaluated the validation-selected checkpoint on the test participants of each fold and reported the mean ± standard deviation across five folds. After fixing the final model configuration, we evaluated it once on the independent holdout set and reported those results separately. We excluded masked and padded epochs from all metric calculations. For key representation and fusion comparisons, two-sided paired t-tests were performed on fold-wise Macro-F1 values from the same five subject-independent cross-validation folds, with p<0.05 considered statistically significant. 4. Results and Discussion In this section, we present the experimental results and discuss the main findings in relation to the study objectives and existing literature. 4.1. Accelerometry Representation Analysis The objective of this experiment was to determine whether wrist accelerometry representation affects sleep-stage classification. We specifically examined whether preserving directional information from the three accelerometer axes provides an advantage over simpler representations of overall movement. We compared three STFT-based ACC representations: axis-averaged, magnitude-based, and separate triaxial representations. We generated all three representations using the same preprocessing and STFT parameters and evaluated them with the same model configuration. This controlled comparison allowed us to isolate the effect of ACC representation on classification performance. Table 6 demonstrates the performance of the three ACC representations for the four-class sleep-staging tasks. ACC magnitude achieved the highest Macro-F1 of 0.4248 and Cohen’s κ of 0.2997. Retaining the three axes separately did not improve class-balanced performance (Macro-F1 = 0.4096), while averaging the axis spectra achieved a Macro-F1 of 0.4185. These results indicate that preserving axis-specific directional information did not provide an additional advantage for sleep-stage classification in this setting. ACC magnitude was therefore selected for the subsequent experiments because it provided the strongest overall class-balanced performance while offering a compact representation that is less dependent on device orientation. 4.2. Architecture Ablation for Handcrafted Features This experiment aimed to select an appropriate architecture for the handcrafted feature representation. We compared conventional machine-learning and temporal deep learning models using the same 112-dimensional feature set and subject-independent five-fold evaluation protocol. As shown in Table 7, Random Forest and RF–HMM achieved the highest overall accuracy, with RF–HMM reaching 0.6647. However, their Macro-F1 and Cohen’s κ were substantially lower than those of the temporal deep learning models. CNN–BiGRU achieved the highest Macro-F1 (0.4251) and Cohen’s κ (0.3475), while maintaining a comparable accuracy of 0.6612. These results show that CNN–BiGRU provided the best class-balanced performance among the evaluated architectures. Its combination of local feature extraction and temporal modeling may help capture patterns across successive sleep epochs. This also suggests that temporal modeling is more important for this task than relying on additional non-temporal classifiers. We therefore selected CNN–BiGRU for the subsequent handcrafted-feature experiments. 4.3. Feature Representation Comparison This experiment aimed to determine how feature representation affects wearable sleep-stage classification. We compared handcrafted, STFT-based, and hybrid representations using the same BVP and ACC signals across the 3-, 4-, and 5-class tasks. Table 8 shows that STFT provided better class-balanced performance than handcrafted features. For the 3-class task, STFT increased accuracy from 68.16% to 73.65%, Cohen’s κ from 0.3271 to 0.4788, and Macro-F1 from 0.5057 to 0.6404. For the 4- and 5-class tasks, handcrafted features achieved slightly higher accuracy than STFT, but STFT produced substantially higher Cohen’s κ and Macro-F1. In the 5-class task, for example, Macro-F1 increased from 0.3366 to 0.4376 despite a small decrease in accuracy. The improvement in Macro-F1 suggests that the STFT representation preserves discriminative time–frequency information that manually designed features may not fully capture. This observation is consistent with previous studies showing that time–frequency representations and data-driven feature learning can provide effective representations for automatic sleep-stage classification [64,65,66]. The hybrid representation achieved the best overall performance across all three tasks. Compared with STFT alone, fusion increased Macro-F1 from 0.6404 to 0.6476 for the 3-class task, from 0.5083 to 0.5108 for the 4-class task, and from 0.4376 to 0.4464 for the 5-class task. These modest gains suggest that STFT already captures most of the discriminative information in BVP and ACC, while handcrafted features provide additional complementary information. Overall, performance decreased as the number of classes increased, reflecting the greater difficulty of distinguishing individual sleep stages. Similar challenges in separating fine-grained sleep stages have been reported in previous wearable sleep-staging studies [67,68,69]. These results support STFT as the primary representation for BVP and ACC, while hybrid fusion provides a small but consistent additional benefit. 4.4. Handcrafted Modality Contribution Across Sleep-Stage Granularities This analysis aimed to determine which wearable modalities provide the most useful handcrafted features and whether combining additional modalities improves sleep-stage classification. We evaluated individual and multimodal combinations of BVP, PRV, HR, ACC, EDA, and temperature across the 3-, 4-, and 5-class tasks. Table 9, Table 10 and Table 11 show that adding more modalities did not consistently improve class-balanced performance. Among the individual modalities, HR achieved the highest Macro-F1 for both the 3-class (0.6011) and 4-class (0.4510) tasks. HR also provided the strongest REM classification, with F1 scores of 0.4315 and 0.4174, respectively. In contrast, ACC was particularly informative for Wake, indicating that cardiac and movement features capture complementary aspects of sleep physiology. The 5-class task was more challenging. The combination of BVP, PRV, HR, ACC, and EDA achieved the highest Macro-F1 of 0.3679 and Cohen’s κ of 0.3392. However, N1 and N3 remained substantially more difficult to classify than Wake, N2, and REM. This difficulty may reflect both their lower representation and their physiological similarity to neighboring sleep stages. Overall, the handcrafted analysis shows that different modalities contribute to different sleep stages. Cardiac features are particularly useful for REM, while ACC provides complementary information for Wake. Importantly, adding more modalities does not necessarily improve performance, indicating that the relevance of the selected physiological signals is more important than the number of modalities. The class-specific results further show that performance varied considerably across sleep stages. NREM in the 3-class task and Light sleep in the 4-class task achieved consistently high F1 scores across most feature configurations. For example, NREM F1 remained around 0.75–0.80 in the 3-class task. Their stronger performance may partly reflect their greater representation in the dataset, which provides more training examples, rather than physiological separability alone. The differences became more pronounced in the 5-class task. BVP + PRV + HR + ACC + EDA achieved the highest overall Macro-F1 of 0.3679 and Cohen’s κ of 0.3392, but N1 and N3 remained difficult to classify. In contrast, N2 showed consistently stronger performance. This pattern may reflect both class imbalance and the physiological similarity of N1 and N3 to neighboring stages. Overall, the results show that each wearable modality contributes differently across sleep stages. Cardiac features were particularly informative for REM, while ACC contributed more strongly to Wake detection. Thus, modality selection should consider stage-specific information rather than simply increasing the number of physiological signals. 4.5. STFT-Based Modality Contribution Across Sleep-Stage Granularities This analysis aimed to determine the individual and complementary contributions of BVP and ACC in the time–frequency domain. We compared BVP alone, ACC magnitude alone, and their combination across the 3-, 4-, and 5-class sleep-stage tasks. Table 12, Table 13 and Table 14 show that BVP was consistently more informative than ACC alone. In the 3-class task, BVP achieved a Macro-F1 of 0.6255, compared with 0.4817 for ACC. Combining both modalities further increased Macro-F1 to 0.6404. The largest modality difference occurred for REM, where BVP achieved an F1-score of 0.4442 compared with only 0.0758 for ACC. The combined model slightly improved REM F1 to 0.4631. A similar pattern appeared in the 4-class task. BVP + ACC achieved the highest Macro-F1 of 0.5083, compared with 0.4996 for BVP and 0.4248 for ACC. However, BVP alone remained slightly better for REM (0.4543 versus 0.4493). These results suggest that ACC provides complementary movement information, particularly for Wake and NREM discrimination, whereas BVP carries stronger information for REM. The pattern changed in the 5-class task. As shown in Table 14, BVP alone achieved the highest Macro-F1 of 0.4520, compared with 0.4376 for BVP + ACC and 0.3731 for ACC. BVP also performed best for N2, N3, and REM, while adding ACC produced only small improvements for Wake and N1. Thus, the benefit of accelerometry decreased when NREM was divided into N1, N2, and N3. The stronger contribution of BVP, particularly for REM, is consistent with the autonomic cardiovascular changes associated with REM sleep [22]. Changes in heart rate and beat-to-beat variability alter the peripheral pulse waveform, and the STFT representation can capture their temporal and spectral dynamics. In contrast, ACC primarily reflects body movement and may provide less stage-specific information during sleep. Overall, BVP was the stronger individual STFT modality, while ACC provided complementary information mainly in the 3- and 4-class tasks. The results also show that combining modalities does not always improve performance as sleep-stage classification becomes more fine-grained. Previous studies have reported benefits from ECG-to-PPG transfer learning [18,19]; therefore, cardiovascular pretraining may provide an avenue to further improve the BVP representation. 4.6. Fusion of STFT and Handcrafted Feature Representations The objective of this experiment was to determine whether handcrafted features provide complementary information beyond the BVP–ACC STFT representation. We fused the STFT model with handcrafted features from individual and combined modalities and evaluated the resulting models across the 3-, 4-, and 5-class tasks. Table 15, Table 16 and Table 17 show modest but consistent benefits from selected fusion configurations. For the 3-class task, fusing handcrafted BVP + ACC with STFT produced the best performance, increasing Macro-F1 from 0.6404 to 0.6476 and accuracy from 73.65% to 75.17%. For the 4-class task, BVP + PRV + HR + Temperature achieved the highest Macro-F1 of 0.5178, compared with 0.5083 for STFT alone. In the 5-class task, BVP + ACC again produced the highest Macro-F1, increasing it from 0.4376 to 0.4464. These results indicate that handcrafted features provide complementary information, but their contribution depends on the classification task. Importantly, adding more modalities did not consistently improve Macro-F1. For example, the full multimodal combination achieved the highest accuracy and Cohen’s κ in the 4-class task, but not the highest Macro-F1. This pattern suggests that the additional modalities improved performance for the more frequent classes without producing equally balanced gains across all sleep stages. EDA and temperature showed smaller and less consistent contributions. However, temperature was useful in the 4-class task, where BVP + PRV + HR + Temperature achieved the highest Macro-F1. This supports previous studies showing that EDA and peripheral temperature contain sleep-related physiological information [3,16,21], although their contribution appears to be task-dependent. Overall, STFT captured most of the discriminative information available from BVP and ACC, while handcrafted features provided a small complementary benefit. The best fusion configuration changed with sleep-stage granularity, showing that adding more modalities is not necessarily more effective than selecting physiologically relevant features for the target task. Paired fold-wise analysis further showed that the STFT BVP + ACC representation significantly outperformed both the corresponding handcrafted BVP + ACC representation (t(4)=9.02, p<0.001) and the best-performing handcrafted single modality, HR (t(4)=3.60, p=0.0227). In contrast, the numerical gains obtained through frozen score-level fusion were not statistically significant. For BVP + ACC, Macro-F1 increased from 0.5083 to 0.5108 (t(4)=2.06, p=0.108), while the best score-level fusion configuration reached 0.5178 (t(4)=1.66, p=0.173). These findings support the advantage of STFT over handcrafted representations while indicating that the relatively small fusion gains should be interpreted cautiously. To further examine the error pattern of the final five-class BVP + ACC fusion model, Figure 4 shows the row-normalized confusion matrix obtained from pooled out-of-fold test predictions. N2 and Wake were classified more reliably, while N1 and N3 remained more difficult and were frequently confused with N2. 4.7. Fusion Strategy Ablation This experiment aimed to determine whether adaptive fusion could improve on simple frozen concatenation. We compared three strategies: frozen concatenation, joint fine-tuning of both encoders, and gated residual fusion. For gated fusion, the STFT and handcrafted embeddings were projected into a common 64-dimensional space, and the handcrafted contribution was controlled by a learned sigmoid gate:zfusion=s+g⊙h, where s and h are the projected STFT and handcrafted embeddings, respectively, and g is the learned gating coefficient. As shown in Table 18, frozen concatenation achieved the highest Macro-F1 of 0.5144±0.0337. Joint fine-tuning and gated fusion increased accuracy to 0.6949 and 0.6940, respectively, but reduced Macro-F1 to 0.4908 and 0.4993. The gated model learned a mean coefficient of 0.477±0.073, indicating that it continued to incorporate the handcrafted representation. Overall, increasing fusion complexity did not improve class-balanced performance. Although the adaptive strategies achieved higher accuracy, frozen concatenation produced the highest Macro-F1 and comparable Cohen’s κ. We therefore retained frozen concatenation as the final fusion strategy. 4.8. MESA Evaluation: Effect of Acquisition Setting and Cohort Size The objective of the MESA evaluation was to examine whether pulse-based sleep-stage classification remained effective under a different acquisition setting and whether increasing the training cohort improved performance. We first evaluated MESA-100, a 100-subject subset selected to provide a cohort more comparable in size and apnea-severity distribution to DREAMT. We then trained the same approach on the full MESA cohort to assess the benefit of a larger, more diverse training population. 4.8.1. MESA-100 Comparison MESA-100 achieved stronger class-balanced performance than the corresponding DREAMT pulse-based experiments across the 3-, 4-, and 5-class tasks. This result suggests that the acquisition setting can influence the sleep-related information captured by peripheral pulse signals. DREAMT acquired BVP from the wrist, whereas MESA acquired PPG using a finger pulse oximeter. However, this comparison should not be interpreted as evidence that finger PPG is inherently superior to wrist BVP. The datasets also differ in acquisition hardware, original sampling rate, cohort characteristics, and other recording conditions. Therefore, the MESA-100 results indicate that acquisition conditions may influence performance, but this comparison cannot isolate the contribution of individual factors. 4.8.2. Effect of Cohort Size We next compared MESA-100 with the full MESA cohort to examine the effect of increasing the training population. As shown in Table 19, performance improved consistently across all three classification tasks. Macro-F1 increased from 0.7059 to 0.8430 for the 3-class task, from 0.6564 to 0.7101 for the 4-class task, and from 0.5515 to 0.6265 for the 5-class task. Cohen’s κ showed the same overall trend. These results demonstrate a clear benefit from increasing the number and diversity of subjects available for training. Nevertheless, performance still decreased as the classification task became more fine-grained. Thus, additional training data improved the robustness of PPG-based sleep staging but did not eliminate the difficulty of distinguishing individual sleep stages. 4.8.3. Stage-Specific Performance and Cross-Dataset Limitations The class-specific results provide further insight into how cohort size affects sleep-stage performance. Figure 5 compares stage-wise F1 scores for the three- and four-class settings. In general, the Full MESA cohort improved performance across most stages, although the magnitude of improvement varied by stage. In the four-class setting, Deep sleep showed a slight decrease despite the larger training cohort. In the 5-class task, the corresponding stage-wise F1 scores are reported in Table 20. Wake, N2, and REM reached F1 scores of 0.8622, 0.6811, and 0.7443 with the full cohort. N1 and N3 remained considerably lower at 0.3624 and 0.4823, respectively. The persistent difficulty of N1 and N3 is particularly important because a similar pattern was observed in DREAMT. Their lower performance therefore persisted across different cohorts, pulse-acquisition locations, and training-set sizes. Although class imbalance may contribute to this difficulty, it is unlikely to fully explain it. N1 is a transitional stage with overlapping characteristics and potentially greater annotation uncertainty, while epoch-level signal artifacts may further affect wearable measurements. In addition, the electrophysiological features used to define NREM stages in PSG may not be fully represented in peripheral cardiovascular signals. Overall, the MESA experiments provide two main findings. First, the stronger MESA-100 results suggest that pulse acquisition and other dataset-specific conditions can influence sleep-stage classification performance. Second, increasing the training population substantially improves PPG-based classification, but does not fully resolve fine-grained NREM staging. The consistent difficulty of N1 and N3 across DREAMT and MESA therefore suggests a broader limitation of peripheral wearable signals for reproducing the distinctions between neighboring NREM stages. 4.9. Limitations This study has several limitations. First, we did not perform direct cross-dataset generalization by training on DREAMT and testing the same model on MESA or another independently collected dataset. This remains an important direction for future work. Second, the analysis was based on existing research datasets and did not include real-world consumer wearable recordings or performance stratification by clinical subgroups such as OSA severity. Third, N1 and N3 showed lower classification performance than the other sleep stages, which may be related to class imbalance, stage-transition ambiguity, and the limited information available from peripheral wearable signals. Finally, the current study evaluated STFT as the main time–frequency representation and did not compare it with other multi-resolution methods. Future work will investigate alternatives such as wavelet-based scalograms. 4.10. Future Work and Clinical Implications Future work will compare STFT with wavelet-based scalograms and other multi-resolution representations. We also plan to explore imbalance-aware learning strategies to improve N1 and N3 classification and to evaluate the framework on additional independent wearable datasets. From a clinical perspective, wearable sleep staging may be useful for long-term sleep monitoring and screening. However, the lower performance for fine-grained stages such as N1 and N3 suggests that these systems should complement, rather than replace, PSG-based clinical assessment. 5. Conclusions In this study, we investigated how signal representation, wearable modality selection, multimodal fusion, acquisition setting, and training cohort size influence wearable sleep-stage classification across three-, four-, and five-class settings. Among the evaluated modalities, cardiovascular signals provided the strongest stage-discriminative information, particularly for REM, while accelerometry contributed complementary movement information, especially for Wake detection. STFT-based representations generally provided stronger class-balanced performance than handcrafted features, whereas combining the two representations yielded only modest additional gains. Importantly, increasing the number of modalities or introducing more complex fusion strategies did not consistently improve Macro-F1, indicating that additional model and sensor complexity is beneficial only when it contributes complementary stage-specific information. The MESA experiments further demonstrated that acquisition conditions and training cohort size can substantially influence performance. Increasing the number and diversity of training subjects improved classification across all sleep-stage granularities; however, N1 and N3 remained comparatively difficult across both DREAMT and MESA. These findings indicate that fine-grained NREM discrimination remains a persistent challenge for sleep staging based on peripheral wearable signals. Overall, the results suggest that effective wearable sleep staging depends more on selecting informative physiological signals and appropriate representations than on simply increasing the number of sensors or model complexity.