Smooth Regularization for Efficient Video Recognition

Gil Goldman, Raja Giryes, Mahadev Satyanarayanan · 2025

Read Smooth Regularization for Efficient Video Recognition by Gil Goldman, Raja Giryes, Mahadev Satyanarayanan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-20928.txt.

Introduction

Gil Goldman, Raja Giryes, and Mahadev Satyanarayanan add a 2025 training trick that makes small video-recognition models much better at reading actions. Video is naturally smooth: what a network sees in one frame should resemble the next. Their method penalises abrupt jumps between the internal representations of consecutive frames, treating frame-to-frame change as a Gaussian random walk, a drift with random steps.

The penalty pushes models toward low-acceleration solutions that match how real motion unfolds. On the Kinetics-600 action benchmark, lightweight models trained this way gained 3.8 to a full 6.4 percentage points of accuracy. The MoViNets family set new state-of-the-art results within their compute budgets, improving by 3.8 to a strong 6.1 points. One model, MoViNet-A3 with the smoothing, reached 85.6 percent accuracy at just 56.4 billion operations, while its closest rival needed 18.3 times more computation. MobileNetV3 and the streaming MoViNets gained 4.9 to nearly 6.4 points at equal memory.

Catalog record

CREATORGil Goldman, Raja Giryes, Mahadev Satyanarayanan
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE33 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-20928

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Smooth Regularization for Efficient Video Recognition Gil Goldman, Raja Giryes, Mahadev Satyanarayanan Abstract We propose a smooth regularization technique that instills a strong temporal inductive bias in video recognition models, particularly benefiting lightweight architectures. Our method encourages smoothness in the intermediate-layer embeddings of consecutive frames by modeling their changes as a Gaussian Random Walk (GRW). This penalizes abrupt representational shifts, thereby promoting low-acceleration solutions that better align with the natural temporal coherence inherent in videos. By leveraging this enforced smoothness, lightweight models can more effectively capture complex temporal dynamics. Applied to such models, our technique yields a 3.8 % 3.8\% – 6.4 % 6.4\% accuracy improvement on Kinetics-600. Notably, the MoViNets model family trained with our smooth regularization improves the current state-of-the-art by 3.8 % 3.8\% – 6.1 % 6.1\% within their respective FLOP constraints, while MobileNetV3 and the MoViNets-Stream family achieve gains of 4.9 % 4.9\% – 6.4 % 6.4\% over prior state-of-the-art models with comparable memory footprints. Our code and models are available at https://github.com/gilgoldm/grw-smoothing . 1 Introduction Video recognition has rapidly evolved over the past decade, with models becoming increasingly capable of learning sophisticated spatial and temporal representations. However, despite remarkable advancements, many architectures still suffer from overfitting or inefficient use of temporal information 7 ; 12 ; 28 . In light of these challenges, we introduce a novel smooth regularization approach designed to instill a strong inductive bias specifically tailored to video data. The key insight behind our method is that video content often exhibits continuous motion and gradual changes in appearance, suggesting that representations should vary smoothly over time. By explicitly encouraging this smoothness, we aim to guide neural networks toward more stable and generalizable internal feature representations, ultimately leading to improved performance across a range of video recognition tasks. Our regularization strategy focuses on the intermediate-layer embeddings produced by a neural network when processing consecutive frames. Instead of allowing these embeddings to fluctuate arbitrarily across frames, we constrain their dynamics to resemble a Brownian motion, which translates to imposing Gaussian Random Walk (GRW) behavior in the frames discrete settings that promotes continuous and relatively modest rates of change. The inspiration behind this modeling choice comes from the fact that, in a typical video, adjacent frames exhibit only gradual shifts in object positioning, scale, lighting, or motion. By treating frame-to-frame representation shifts as a form of GRW, we incorporate a principled, mathematically grounded way to preserve smoothness in the learned embeddings. This not only reflects a more natural representation of videos but also acts as a regularizing force against abrupt or erratic changes in the network’s internal states. A core outcome of our approach is that it naturally discourages large jumps between successive embeddings. Mathematically, our formulation involves adding the GRW penalty term to the training objective, which grows whenever the model produces excessively rapid transitions between frames in the network embedding space. By penalizing these abrupt shifts, we encourage networks to learn features that evolve more gently over time. This favors solutions that maintain a sense of temporal consistency, that is, low acceleration in the embedding space, leading to more coherent internal representations that can better capture the true temporal dynamics present in real-world videos. By aligning model training with the natural temporal structure found in the data, our approach makes it easier to networks to focus on learning meaningful temporal correlations rather than wasting capacity on fitting noisy or abrupt changes. Consequently, the model becomes more sensitive to subtle motion cues, often crucial for recognizing fine-grained actions or micro-movement semantics without sacrificing robustness to variations due to noise in the network embedding space. While bigger network may have enough capacity to learn both the variations and noise in the embedding space together with the changes in motion, this is more challenging in resource constrained networks. Thus, we focus in this work on such type of networks. We demonstrate the effectiveness of our proposed smooth regularization on such lightweight models. To verify its benefits, we trained these smaller-scale networks on the popular Kinetics-600 dataset, a large benchmark known for its diversity of human action classes. By simply adding our novel loss function to the training of these architectures, we get consistent gains of 3.8\% – 6.4\% in classification accuracy as shown in Figure 1 , leading to new state-of-the-art performance under FLOP and memory constraints. Figure 1 : Performance Results on Kinetics-600. By simply adding GRW-smoothing to existing models, we achieve significant improvements. Left: Accuracy vs. FLOPs, where each point corresponds to a published model (see Table 1 for references). GRW-smoothing improves the state-of-the-art performance of efficient models by 3.8–6.1%. Notably, MoViNet-A3-GRW achieves 85.6% accuracy at just 56.4 GFLOPs, while the closest model, MViTv2-B-32×3, requires 18.3 \times more FLOPs. Right: Accuracy vs. Memory. GRW -smoothing improves the state-of-the-art performance of memory-efficient models by 4.9–6.4%. Our main contributions can be summarized as follows: Smoothness Prior in Video Recognition : We introduce a novel regularization technique that enforces smoothness in the intermediate-layer embeddings of consecutive video frames by modeling their changes as a GRW. State-of-the-Art Performance Under Efficiency Constraints : Our technique outperforms current leading solutions within a similar memory and compute range, confirming the broad applicability of our method to resource-limited scenarios. Flexible Framework : Focusing on smoothness as a strong inductive bias provides a plug-and-play regularization option for existing video recognition pipelines. It integrates seamlessly into different architectures with a negligible computational overhead as illustrated in Figure 4 . 2 Related Work Video recognition in general. Early approaches to video action recognition learned spatiotemporal representations with 3D CNNs 5 ; 27 , with subsequent advances such as Two-Stream networks 26 and SlowFast 8 improving the balance between spatial semantics and motion modeling. Temporal segment sampling 30 provided an efficient way to cover long videos with sparse clips. With the advent of transformers, attention-based models extended self-attention to spatiotemporal tokens 2 ; 1 . Subsequent work introduced hierarchical and multiscale designs and localized attention to improve efficiency 6 ; 22 , and explored alternative attention mechanisms such as trajectory attention 24 , as well as MLP-like backbones 32 and video-specific ViT adaptations 25 . Despite strong accuracy, pure transformers often incur substantial computational and memory costs for long videos, particularly relative to compact CNN baselines 2 ; 1 ; 22 . Lightweight video recognition. A parallel line of work targets real-time and on-device deployment by reducing compute and parameters. CNN-based designs dominate this space due to depthwise separable convolutions and efficient temporal operators. MoViNets 15 introduce a NAS - designed family of efficient 3D CNNs and a stream buffer for constant - memory streaming, achieving strong accuracy–efficiency trade - offs. X3D 9 systematically compounds network width, depth, and temporal resolution to yield compact 3D CNNs. Temporal Shift Module (TSM) 21 augments 2D CNN backbones with a zero - parameter, zero - FLOP temporal exchange, enabling video recognition with image - classification - level compute. Further lightweight temporal modules include TEA 19 , TDN 29 , and TAdaConv 13 , which inject temporal cues with modest overhead. Transformer-based models such as MViT 6 and VideoSwin-T 22 reduce attention cost via multiscale hierarchies or windowed attention, yet they often remain heavier than the most compact CNN baselines

Read the complete text