Respiratory Status Detection with Video Transformers

Thomas Savage, Evan Madill · 2026

Read Respiratory Status Detection with Video Transformers by Thomas Savage, Evan Madill (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-21349.txt.

Introduction

Savage and Madill test whether video transformers can detect respiratory distress from video. Clinicians spot early signs of breathing decline by sight, which opens a window for earlier care. They filmed healthy volunteers recovering after hard exercise. Early clips show more shortness of breath, which creates a temporal ordering task. A ViViT encoder with Lie relative encodings and motion guided masking, plus an embedding comparison strategy, reaches an F1 score of zero point eight one. The results suggest modern video transformers can read subtle changes in breathing mechanics.

Catalog record

CREATORThomas Savage, Evan Madill
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE17764
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.21349

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Respiratory Status Detection with Video Transformers Thomas Savage, Evan Madill Abstract Recognition of respiratory distress through visual inspection is a life-saving clinical skill. Clinicians can detect early signs of respiratory deterioration, creating a valuable window for earlier intervention. In this study, we evaluate whether recent advances in video transformers can enable Artificial Intelligence systems to recognize the signs of respiratory distress from video. We collected videos of healthy volunteers recovering after strenuous exercise and used the natural recovery of each participant’s respiratory status to create a labeled dataset for respiratory distress. Splitting the video into short clips, with earlier clips corresponding to more shortness of breath, we designed a temporal-ordering challenge to assess whether an AI system can detect respiratory distress. We found a ViViT encoder augmented with Lie Relative Encodings (LieRE) and Motion-Guided Masking, combined with an embedding-based comparison strategy, can achieve an F1 score of 0.81 on this task. Our findings suggest that modern video transformers can recognize subtle changes in respiratory mechanics. \icml@noticeprintedtrue † † footnotetext: \forloop @affilnum1¡ 0 AUTHORERR: Missing \icmlaffiliation. \Notice@String 1 Introduction Respiratory status evaluation is a key skill taught to all nurses, physicians, and paramedics. Signs of distress such as nasal flaring, skin retraction, and the use of accessory muscles can be subtle, but indicate that a patient is at high risk for respiratory failure and even death [ 1 , 2 , 3 ] . Providers become attuned to these signs over years of training, which enables them to look beyond telemetry or laboratory-based measurements of respiratory status and recognize impending decompensation. In turn, frequent monitoring of high-risk patients by trained professionals is both a significant safety priority and operational expense for hospitals and skilled nursing facilities [ 4 , 5 , 6 , 7 ] . In this paper, we investigate whether a computer vision application can be trained to recognize the subtle signs of respiratory distress. Historically, computer vision applications have struggled with this task because the detection of small changes in breathing mechanics is confounded by patient motion, body positioning, and variable lighting conditions [ 8 , 9 ] . Encouragingly, recent advances in video transformer positional encoders have improved spatiotemporal modeling and may enable detection of these subtle respiratory changes [ 10 ] . Building on these advances, our objective is to determine whether modern video transformer architectures can detect subtle changes in breathing mechanics. We evaluate this hypothesis in a cohort of healthy volunteers who performed exercise and recorded their respiratory recovery. We trained the model on a pairwise temporal ordering task, predicting which of two short clips from the same recovery sequence occurred earlier (greater respiratory distress) versus later (closer to baseline). Accurate ordering of these clips provides evidence that computer vision models can capture subtle changes in respiratory status. 2 Related Work 2.1 Video Transformer Background Video transformer architectures such as ViViT [ 11 ] and TimeSformer [ 12 ] represent videos as sequences of spatiotemporal patch tokens, using attention to model relationships across space and time for video classification. Because attention alone has no built-in awareness of token position, these models must provide positional information to distinguish the location of tokens. Early video transformers used absolute positional embeddings (APEs), which provide each token a learned position label. A limitation of APEs is that they can lead to position-specific overfitting, thereby degrading performance, especially for tasks that rely on subtle, temporally-evolving visual signals [ 13 ] . To improve performance, relative positional encodings were developed to reduce the risk of overfitting. Rotary Position Embeddings (RoPE) is a widely used relative method that encodes position by applying position-dependent rotations to the query and key vectors, so their dot-products naturally reflect relative token position rather than requiring learned absolute position tokens [ 14 ] . Although RoPE was originally developed for 1D sequences, such as language models, rotary approaches have since been adapted to higher-dimensional vision settings. Lie Relative Encodings (LieRE) further generalizes rotary encodings by learning higher-dimensional rotation bases for multi-axis coordinate systems (e.g., x, y, and time for video data), enabling richer spatiotemporal position representations and significant improvements in performance [ 10 ] . An additional strategy that informed our approach is Motion-Guided Masking (MGM), which encourages the model to focus on regions of the video that exhibit meaningful motion [ 16 ] . In MGM, each frame is divided into a grid of patches, optical flow based motion is computed for each patch, and only the most dynamic patches are retained as input to the model. The remaining patches are masked (blacked out). By restricting focus to the most informative spatiotemporal regions, this motion-guided masking helps the transformer down-weight irrelevant visual content (e.g., background, lighting variation, or non-respiratory movement) and can improve performance on tasks that depend on subtle temporal changes [ 15 , 16 ] . 2.2 Respiratory Distress Assessment via Computer Vision Most published computer vision methods for respiratory monitoring focus on estimating respiratory rate or tidal volume, rather than detecting the classic visual physical exam signs of respiratory distress [ 8 , 9 , 17 ] . In our literature review, we identified only one study—Nawaz et al. [ 18 ] —that explicitly targeted physical exam signs of dyspnea. In their work, the authors used a 3D convolutional neural network to identify chest retractions and detect acute respiratory distress in children admitted to a pediatric intensive care unit. A key limitation of the Nawaz et al. approach was its reliance on manual video cropping and clinician-guided positioning prior to recording [ 18 ] . While this added supervision can improve signal quality, it reduces scalability in real-world monitoring environments, where deployment typically requires minimal curation and limited dependence on human-in-the-loop preprocessing. In contrast, our experiment focuses on fully automated video processing without manual cropping or clinician-guided framing. This design choice prioritizes practicality for routine monitoring settings and is better suited for scalable deployment. 3 Methods In this study, we evaluate respiratory status change detection through a pairwise temporal ordering objective. Given two short video clips sampled from the same participant’s post-exercise recovery recording, the model predicts which clip occurred earlier in the sequence—corresponding to greater shortness of breath—versus later, when the participant is closer to their baseline. To solve this challenge, we evaluated multiple strategies for (1) comparing clip states and (2) capturing spatiotemporal features for best performance. 3.1 Clip-State Comparison Because our experiment compares respiratory states between two clips, rather than performing standalone classification on a single clip, we required an explicit mechanism to compare two videos. We evaluated two primary approaches: • Two-tower analysis with cross-attention • Mapping each video to an embedding state for comparison The two-tower cross-attention approach processed the two clips through separate model towers, producing two feature representations. These representations were then contrasted using cross-attention, enabling direct interaction between clip representations. We tested two cross-attention configurations: (1) full cross-attention across all layers and (2) cross attention at the level of the CLS-token. The model was trained to predict which of the two clips depicted greater shortness of breath (i.e., whether the first or second clip occurred earlier in the recovery sequence). In our embedding-state approach, a single model encoded each clip into a fixed-length embedding

Read the complete text