FrameSkip: Learning from Fewer but More Informative Frames in VLA Training
Bin Yu, Shijie Lian, Xiaopeng Lin, Zhaolong Shen, Yuliang Wei, Changti Wu, Hang Yuan, Haishan Liu, Bailing Wang, Cong Huang, Kai Chen · 2026
Read FrameSkip: Learning from Fewer but More Informative Frames in VLA Training by Bin Yu, Shijie Lian, Xiaopeng Lin, Zhaolong Shen, Yuliang Wei, Changti Wu, Hang Yuan, Haishan Liu, Bailing Wang, Cong Huang, Kai Chen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-13757.txt.
Introduction
Bin Yu and ten colleagues introduce FrameSkip, a data-layer frame picker for VLA training. It scores path frames by action shifts, visual-action match, task-progress priors and gripper-shift keeping. FrameSkip remaps training samples toward high-value frames under a target keeping ratio, working only in the loader without changing design, action head, goal or use. Across RoboCasa-GR1, SimplerEnv and LIBERO, FrameSkip reaches 76.15 percent macro-mean success versus 66.50 percent for full-frame training, keeping 20 percent of unique frames.
Catalog record
| CREATOR | Bin Yu, Shijie Lian, Xiaopeng Lin, Zhaolong Shen, Yuliang Wei, Changti Wu, Hang Yuan, Haishan Liu, Bailing Wang, Cong Huang, Kai Chen |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 38540 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.13757
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
FrameSkip: Learning from Fewer but More Informative Frames in VLA Training Bin Yu, Shijie Lian, Xiaopeng Lin, Zhaolong Shen, Yuliang Wei, Changti Wu, Hang Yuan, Haishan Liu, Bailing Wang, Cong Huang, Kai Chen Abstract Vision-Language-Action (VLA) policies are commonly trained from dense robot demonstration trajectories, often collected through teleoperation, by sampling every recorded frame as if it provided equally useful supervision. We argue that this convention creates a temporal supervision imbalance: long low-change segments dominate the training stream, while manipulation-critical transitions such as alignment, contact, grasping, and release appear only sparsely. We introduce FrameSkip , a data-layer frame selection framework that scores trajectory frames using action variation, visual-action coherence, task-progress priors, and gripper-transition preservation, then remaps training samples toward high-importance frames under a target retention ratio. Because FrameSkip operates only in the dataloader, it leaves the VLA architecture, action head, training objective, and inference procedure unchanged. Across RoboCasa-GR1, SimplerEnv, and LIBERO, FrameSkip improves the success-retention trade-off over full-frame training and simpler frame selection variants, achieving a macro-average success rate of 76.15% across the three benchmarks compared with 66.50% for full-frame training while using a compressed trajectory view that retains 20% of unique frames in the main setting. Code and model checkpoints are available on GitHub and Hugging Face . 3 3 footnotetext: Work done at Zhongguancun Academy (Beijing). 1 Introduction Vision-Language-Action (VLA) models have recently emerged as a promising paradigm for robotic manipulation by combining visual grounding, language conditioning, and action prediction within a unified policy model ( 31 ; 10 ; 2 ; 36 ) . As these systems scale to broader data mixtures, more tasks, and stronger vision-language backbones, they are increasingly trained on large embodied datasets such as Open X-Embodiment ( 21 ) . These datasets are typically composed of dense robot demonstration trajectories, often collected through teleoperation, where each trajectory records a sequence of observations and actions produced while completing a task. This scaling trend has improved task coverage and generalization, but it also exposes a basic training convention that remains largely unquestioned: dense demonstrations are sampled as if every trajectory frame provided equally useful supervision. Figure 1: FrameSkip reframes training-time frame pruning as temporal supervision allocation: it reduces exposure to redundant low-change trajectory segments and increases exposure to manipulation-critical transitions. This convention is mismatched with the temporal structure of robot demonstrations, as illustrated in Figure 1 . Manipulation trajectories often contain long low-change segments, such as approaching an object, maintaining a grasp, or transporting an object steadily toward a target. In contrast, the moments that define the task outcome are sparse: alignment, contact, grasp closure, release, and abrupt changes in end-effector behavior may occupy only a small fraction of the recorded trajectory. Uniform frame sampling therefore creates a temporal supervision imbalance. Under a fixed optimization budget, rare decision-critical transitions can be diluted by abundant but weakly informative observations. As illustrated in Figure 2 , failures are not uniformly distributed along a trajectory: routine stages such as approach and return are often handled reliably, whereas sparse interaction stages such as alignment, grasping, and release exhibit substantially higher failure rates. This stage-wise failure concentration suggests that VLA policies can adapt to dominant smooth motions while remaining brittle at sparse manipulation-critical transitions. We interpret this pattern as global adaptation but local under-supervision, motivating frame selection not as data reduction alone, but as a way to rebalance training toward the moments where policy learning is most fragile. Existing VLA research has largely addressed scaling through model architecture, action representation, data mixture design, and optimization strategy ( 10 ; 9 ; 23 ; 8 ; 19 ) . Much less attention has been paid to how supervision is distributed across the frames within each demonstration. Yet this frame-level structure is especially important in embodied data, where trajectories are temporally dense, physically constrained, and dominated by smooth motion. This raises a simple question: can VLA training benefit from reallocating supervision toward the frames that carry the most policy-relevant information? Figure 2: Robot trajectories contain long redundant segments and sparse manipulation-critical transitions, motivating frame selection as a training supervision allocation problem. We therefore view frame selection not merely as a way to reduce data volume, but as a mechanism for reallocating temporal supervision under a fixed optimization budget. In this paper, we present FrameSkip , a data-layer frame selection framework for VLA training. FrameSkip assigns each frame an importance score from lightweight trajectory cues, including action variation, visual-action coherence, task-progress priors, and gripper-transition preservation. It then constructs compressed trajectory views under target retention ratios and remaps training samples toward retained high-importance frames. Importantly, FrameSkip does not modify the VLA architecture, action head, loss function, or inference procedure. This makes FrameSkip a direct way to study frame importance as a training principle rather than as a model-specific architectural change. We evaluate FrameSkip as a question about the success-retention trade-off of VLA training rather than as a generic frame dropping heuristic. Under matched settings, we compare full-frame training, random frame selection, action-variation-only selection, and progressively stronger importance metrics on RoboCasa-GR1 ( 18 ) , SimplerEnv ( 13 ) , and LIBERO ( 16 ) . In the main setting, FrameSkip uses a compressed trajectory view that retains 20% of unique frames and improves the macro-average success rate across the three benchmarks from 66.50% with full-frame training to 76.15%, with consistent gains on all three benchmarks. Our main contributions are as follows: • To our knowledge, we present the first VLA training approach that optimizes supervision at the frame level, identifying temporal supervision imbalance as a practical and underexplored issue in VLA training. • We introduce FrameSkip , an architecture-agnostic data-layer framework that selects more informative training frames using lightweight trajectory cues and gripper-transition preservation. • We provide a systematic empirical study of importance-guided frame retention, including matched-ratio baselines and ablations over retention ratios, importance metrics, and warmup schedules. 2 Related Work Vision-language-action models. VLA models combine visual grounding, language conditioning, and action prediction in a unified policy interface ( 10 ; 2 ; 36 ) . Recent work improves these systems through stronger VLM initialization, action tokenization, diffusion or flow-matching action heads, and large-scale cross-embodiment data ( 23 ; 8 ; 19 ; 21 ) . These advances generally assume that the training set is consumed at its original temporal density. FrameSkip is complementary: it asks whether the same VLA families can be trained with fewer but more informative frames. Data Curation for Robot Learning. Coarse-grained approaches reweight datasets ( 5 ) or filter trajectories 6 but treat intra-trajectory frames uniformly. Scizor 34 curates transitions via a learned task-progress predictor, aiming to remove low-quality and redundant data. FrameSkip differs in objective and mechanism: it does not learn an auxiliary transition-quality model or frame deletion policy, but reallocates training supervision within each trajectory using lightweight cues, including action variation, visual-action coherence, task-progress priors, and gripper-transition preservation, under a controllable retention ratio. TGM-VLA 24 addresses keyframe over-sampling in 3D manipulation, but is specific to keyframe-based architectures. FrameSkip operates on raw frames without keyframe structure. Figure 3: FrameSkip pipeline. FrameSkip scores frames in each demonstration trajectory, retains high-importance frames under a target ratio, and injects the compressed trajectory view into VLA training through dataloader index remapping while