StretchySnake: Flexible SSM Training Unlocks Action Recognition Across Spatio-Temporal Scales
Nyle Siddiqui, Rohit Gupta, Sirnam Swetha, Mubarak Shah · 2025
Read StretchySnake: Flexible SSM Training Unlocks Action Recognition Across Spatio-Temporal Scales by Nyle Siddiqui, Rohit Gupta, Sirnam Swetha, Mubarak Shah (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-16209.txt.
Introduction
Nyle Siddiqui and three co-authors teach a video model to handle clips of any size or length. Their model is a state space model, a design that scans a sequence with a rolling inner state instead of costly attention. Such models are usually trained at one fixed frame size and clip length, and they falter on videos that differ.
Their fix, nicknamed StretchySnake, trains on videos sampled at mixed sizes and lengths. It also reshapes the model's weights on the fly to fit each new scale. Five versions of the recipe were tried, and the best is named. On short-action and long-action test sets, StretchySnake beats both transformer and state space rivals by up to twenty-eight percent. It also keeps its edge on fine, subtle actions, making one model serve many video jobs.
Catalog record
| CREATOR | Nyle Siddiqui, Rohit Gupta, Sirnam Swetha, Mubarak Shah |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2510-16209
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
StretchySnake: Flexible SSM Training Unlocks Action Recognition Across Spatio-Temporal Scales Nyle Siddiqui, Rohit Gupta, Sirnam Swetha, Mubarak Shah Abstract State space models (SSMs) have recently emerged as a competitive alternative to transformers in various linguistic and visual tasks. Their linear complexity and hidden-state recurrence make them particularly attractive for modeling long sequences, whereas attention becomes quadratically expensive. However, current training methods for video understanding are tailored towards transformers and fail to fully leverage the unique attributes of SSMs. For example, video models are often trained at a fixed resolution and video length to balance the quadratic scaling of attention cost against performance. Consequently, these models suffer from degraded performance when evaluated on videos with spatial and temporal resolutions unseen during training; a property we call spatio-temporal inflexibility. In the context of action recognition, this severely limits a model’s ability to retain performance across both short- and long-form videos. Therefore, we propose a flexible training method that leverages and improves the inherent adaptability of SSMs. Our method samples videos at varying temporal and spatial resolutions during training and dynamically interpolates model weights to accommodate any spatio-temporal scale. This instills our SSM, which we call StretchySnake , with spatio-temporal flexibility and enables it to seamlessly handle videos ranging from short, fine-grained clips to long, complex activities. We introduce and compare five different variants of flexible training, and identify the most effective strategy for video SSMs. On 6 6 action video benchmarks, StretchySnake outperforms vanilla VideoMamba by up to 28%, while simultaneously delivering 3x speedups and a 90% reduction in GFLOPs in low-resolution settings. On short-action (UCF-101, HMDB-51) and long-action (COIN, Breakfast) benchmarks, StretchySnake outperforms transformer and SSM baselines alike, with strong adaptability to fine-grained actions (SSV2, Diving-48). Therefore, our method provides a simple drop-in training recipe that makes video SSMs more robust, resolution-agnostic, and efficient across diverse action recognition scenarios. Figure 1: Left: StretchySnake is robust to a wide range of spatio-temporal resolutions, optimizing the accuracy and cost tradeoff depending on the setting. Compared to transformer and SSM baselines, StretchySnake achieves higher accuracy with dramatically lower compute cost. On COIN & Breakfast, StretchySnake achieves up to +4.4% accuracy and > 90% GFLOPs reduction over VideoMamba (dashed line). Right: Across 6 diverse action recognition benchmarks, spanning short, long, and fine-grained motion, StretchySnake consistently achieves the best retrieval performance. 1 Introduction Among the broad spectrum of challenges in video action understanding, there are two attributes that are particularly central to what models usually must capture: the length of the action ( Choudhury et al. 2024 ; Yang et al. 2024 ; Lee et al. 2024b ; Kahatapitiya et al. 2024 ) and its granularity ( Swetha et al. 2021 ; Doughty & Snoek 2022 ; Gupta et al. 2023 ; Zhang et al. 2024b ; Zhang et al. 2024a ; Nie et al. 2024 ) . In the context of action length, a robust action recognition model ideally can capture both short, quick actions that only last a few seconds as well as long, complex activities that unfold over minutes. In terms of action granularity, coarse actions (e.g., jumping vs. waving) can often be recognized from global motion patterns, whereas fine-grained actions (e.g., dropping vs. setting down an object) require more attention to subtle spatio-temporal cues; a robust model should also excel at both of these objectives. We refer to this challenge as unified action recognition , where a single video backbone should perform well across short-length, long-length, coarse-grained, and fine-grained videos alike. This challenge is compounded by the fact that action length and granularity require different input characteristics: accurately recognizing long activities often requires many frames over time ( Zhou et al. 2025 ; Ye et al. 2025 ; Tan et al. 2024 ) , whereas distinguishing fine-grained actions can depend on video length and spatial resolution to capture subtle details ( Li et al. 2022 ; Xu et al. 2024 ) . Furthermore, real-world videos exist at diverse resolutions and durations, ranging from high-resolution sports replays to low-quality surveillance footage. Thus, there is a significant benefit in developing models that adapt seamlessly across diverse frame sizes (spatial resolution) and number of frames (temporal resolution). However, most existing video models are trained in a static fashion ( Arnab et al. 2021 ; Swetha et al. 2021 ; Bertasius et al. 2021 ; Liu et al. 2022 ; Gupta et al. 2024 ; Li et al. 2024 ) , where both the spatial and temporal resolutions are fixed. While this simplifies training, it introduces what we call spatio-temporal inflexibility : a model’s accuracy degrades when evaluated at resolutions or clip lengths unseen during training. For example, previous image-only studies have shown that image models suffer massive performance drops when simply tested at spatial resolutions unseen during training ( Tian et al. 2023 ; Beyer et al. 2023 ) . We are the first to show that this traditional method of training still perpetuates inflexibility in both the learned spatial and temporal features (Fig. 3 ). Moreover, transformers are the dominant backbone for video understanding tasks, including action recognition, ( Siddiqui et al. 2024 ) , object segmentation ( Kirillov et al. 2023 ) , and multi-modal learning ( Zhu et al. ; Lin et al. 2023 ; Swetha et al. 2023 ) . Yet, they face two fundamental limitations in the context of unified action recognition, where a single model must efficiently and robustly scale to a variety of spatio-temporal resolutions. First, their quadratic attention complexity makes them computationally prohibitive for extended activities such as long-activity videos. Secondly, their reliance on learning explicit token-to-token relationships usually constrains them to fixed input sizes, preventing generalization across diverse spatio-temporal scales. Consequently, transformer-based video models are ill-suited as a straightforward solution to unified action recognition: we later show that naively applying spatio-temporal flexible training to popular video transformers yields inconsistent, architecture-dependent gains (Sec. A.2 ). To this end, state space models (SSMs) ( Orvieto et al. 2023 ; Smith et al. 2022 ; Gu et al. 2021b ) are a promising alternative. Unlike attention, SSMs learn to compress sequences into a hidden state and operate at near-linear complexity, which is advantageous for long-range modeling. Moreover, their recurrence-based formulation is more robust to variable input lengths, which could better support a unified action recognition model that scales from short actions to long activities. However, existing video SSMs ( Li et al. 2024 ) follow traditional training practices with fixed spatio-temporal resolutions, which underutilizes their context-dynamic, long-range capabilities and forces them to inherit the same rigidity as transformers . We address this by proposing a flexible training method tailored to unlock the latent potential of SSMs. Instead of training on a fixed resolution and video length, we dynamically sample different spatio-temporal scales during training and interpolate input size-dependent weights on the fly, without architectural changes. Concretely, we interpolate spatial and temporal positional embeddings and the patch-embedding convolution when sampling different heights, widths, and frames while training (Fig. 2 ). This instills our model, StretchySnake , with spatio-temporal flexibility (or st-flexibility) that (1) generalizes across a wide range of scales, and (2) improves action recognition performance. Because spatio-temporal flexibility can be implemented in multiple ways, we systematically introduce and evaluate