Temporal Sampling Frequency Matters: A Capacity-Aware Study of End-to-End Driving Trajectory Prediction

Yumao Liu, Tao Liu, Xiangyu Li, Jiaxiang Li, Ke Ma · 2026

Read Temporal Sampling Frequency Matters: A Capacity-Aware Study of End-to-End Driving Trajectory Prediction by Yumao Liu, Tao Liu, Xiangyu Li, Jiaxiang Li, Ke Ma (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-10388.txt.

Introduction

Yumao Liu and four colleagues question an assumption in self-driving research: that denser camera sampling always helps trajectory prediction. End-to-end driving models usually train on frames at the highest available rate, assuming more frames mean better predictions. Their 2026 paper treats sampling rate as an explicit training-set design variable and sweeps it.

Starting from high-rate datasets, the authors built sweep training sets by subsampling frames along each timeline, then trained and tested the same models under a fixed protocol. Results on Waymo, nuScenes, and PAVE varied by model and dataset. Smaller end-to-end models often peaked at lower or middle rates, while the larger AutoVLA model peaked at the highest rate on all three datasets. Matched-iteration controls suggest the smaller models' edge is not merely an artifact of update counts. Sampling rate, the authors conclude, should be reported and tuned, not left at maximum.

Catalog record

CREATORYumao Liu, Tao Liu, Xiangyu Li, Jiaxiang Li, Ke Ma
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE50 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.10388

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Temporal Sampling Frequency Matters: A Capacity-Aware Study of End-to-End Driving Trajectory Prediction Yumao Liu, Tao Liu, Xiangyu Li, Jiaxiang Li, Ke Ma Abstract End-to-end (E2E) autonomous driving trajectory-prediction is often trained using camera frames sampled at the highest available temporal sampling frequency. This practice implicitly assumes that denser temporal sampling monotonically improves trajectory-prediction performance. This paper questions that assumption by studying temporal sampling frequency as an explicit training-set construction variable. Starting from E2E datasets with high native camera-frame frequencies, we construct frequency-sweep training sets by temporally subsampling camera frames along each trajectory timeline. For each model–dataset pair, we train and evaluate the same model under a fixed protocol, so that the resulting frequency-response reflects how trajectory-prediction performance changes with temporal sampling frequency. We analyze this response from a capacity-aware view: sparse temporal sampling may miss driving-relevant information, whereas dense temporal sampling may introduce redundant visual content and driving-irrelevant off-manifold noise. They may together constitute a driving-irrelevant capacity burden for finite-capacity models. Experiments are conducted on Waymo, nuScenes, and PAVE with three smaller E2E models and a larger VLA-style AutoVLA model. The results show model- and dataset-dependent frequency-responses. Smaller E2E models often exhibit non-monotonic or near-plateau responses and achieve their best 3-second ADE at lower or intermediate temporal sampling frequencies. In contrast, AutoVLA achieves its best 3-second ADE/FDE at the highest evaluated frequency on all three datasets. Iteration-matched controls further suggest that the lower/intermediate-frequency advantage of smaller models is not explained solely by unequal training-update counts. These findings show that temporal sampling frequency should be reported and tuned rather than simply set to the highest available frequency. 1 Introduction End-to-end (E2E) trajectory prediction models learn a direct mapping from sensor and state observations to future ego trajectories for autonomous driving [ 7 , 3 ] . In contrast to modular design (perception, tracking, prediction, and planning stages), an E2E model learns this mapping directly from large-scale human-driving data [ 3 ] . These data are inherently temporal: camera frames, ego-state histories, and other observations are collected along trajectory timelines. Thus, constructing an E2E training set requires deciding how densely camera-frame timestamps should be retained along trajectory. In this paper, we refer to this density as the temporal sampling frequency . It denotes the rate at which camera-frame timestamps are retained when constructing a training set, or equivalently, the inverse of the timestamp gap between consecutive selected camera frames within the same trajectory. Although often treated as a preprocessing detail, this frequency directly controls the density of training samples and the temporal spacing between neighboring observations. In almost all existing E2E studies, the temporal sampling frequency is inherited from the original dataset rather than treated as an explicit design choice. The rationale is straightforward: the dataset-provided frequency is usually the highest readily available frequency after data collection. It is therefore often viewed as a conservative choice that appears to preserve temporal information by retaining more camera-frame timestamps and creating more training samples. For example, many nuScenes-based pipelines use 2 Hz keyframes from the original 12 Hz camera stream, whereas Waymo provides 10 Hz camera frames [ 25 , 30 , 15 ] . However, denser temporal sampling does not necessarily provide proportionally more trajectory-relevant information. Neighboring frames can be highly correlated, and the marginal benefit of additional samples may depend on the dataset dynamics and the model family [ 22 ] . Our motivation is inspired by recent discussions on data structure and model capacity. For example, a recent study argues from a denoising-model perspective that useful data structure can be concentrated near a lower-dimensional manifold, whereas high-dimensional details or noise may impose additional modeling burden without proportional task benefit [ 16 ] . Although temporal sampling in E2E driving is a different problem, this perspective motivates a related question: Does denser camera-frame sampling always provide useful trajectory-relevant evidence for finite-capacity E2E models? We use this question to frame temporal sampling frequency as a capacity-aware problem rather than as a fixed preprocessing choice. As illustrated in Figure 1 , temporal sampling frequency controls a trade-off between driving-relevant information and driving-irrelevant capacity burden. Sparse sampling may miss driving-relevant information, such as ego motion, agent interactions, and scene evolution. Dense sampling provides more camera frames and improves temporal coverage, but adjacent frames may contain highly redundant visual content and driving-irrelevant off-manifold noise, such as sensor jitter, photometric changes, and background motion [ 22 , 26 ] . This trade-off becomes important because E2E trajectory-prediction models have finite capacity. Here, model capacity refers to the ability of a model to represent and learn task-relevant patterns from data. Redundant visual content and driving-irrelevant off-manifold noise may consume part of this capacity while providing limited additional benefit for trajectory prediction [ 16 ] . We summarize these factors as the driving-irrelevant capacity burden . Under this capacity-aware view, increasing temporal sampling frequency may add driving-relevant information, but it may also increase the driving-irrelevant capacity burden. Therefore, the relationship between temporal sampling frequency and trajectory-prediction performance may be non-monotonic, and the best frequency may depend on dataset dynamics and model capacity. This raises a basic but underexplored question: How should temporal sampling frequency be chosen for E2E trajectory prediction? Is the highest available frequency always the best, or does the best frequency depend on the dataset and the model? Figure 1: Capacity-aware view of temporal sampling frequency. Sparse temporal sampling may miss driving-relevant information. Dense temporal sampling can improve coverage of driving-relevant information, but it can also introduce redundant visual content and driving-irrelevant off-manifold noise. With finite model capacity, the best temporal sampling frequency may therefore vary across datasets and models. We study this question by measuring the frequency-response of different E2E trajectory prediction models, defined as the change in trajectory-prediction performance across temporal sampling frequencies. Starting from three E2E datasets, namely Waymo [ 28 ] , nuScenes [ 4 ] , and PAVE [ 17 ] , we construct frequency-sweep training sets by temporally subsampling camera frames along each trajectory timeline. Each training set corresponds to one temporal sampling frequency. Higher-frequency training sets retain more camera frames and generate more training samples, whereas lower-frequency training sets retain fewer frames and generate fewer samples. We refer to the performance variation across these training sets as the frequency-response of an E2E model. We evaluate four E2E trajectory-prediction models: three smaller models with fewer than 20M parameters, namely E2EDriver, BEV-E2EDriver [ 21 ] , and Tiny-SSR [ 15 ] , and one larger vision-language-action(VLA) [ 33 ] style model, AutoVLA [ 32 ] with approximately 3 B parameters. For each model–dataset pair, we keep the model architecture, per-sample input format, future ego-trajectory target, loss function, optimizer, and evaluation protocol fixed, and vary only the temporal sampling frequency used to construct the training set. Since higher-frequency training sets contain more samples, we also conduct iteration-matched controls to examine whether the observed frequency-response is caused solely by unequal training-update counts. The results show that temporal sampling frequency has a nontrivial effect on E2E trajectory-prediction performance. The highest available frequency is not always the best choice, especially for smaller E2E models, which often show non-monotonic or near-plateau frequency responses. In contrast, the larger VLA-style model shows a stronger preference for higher temporal sampling

Read the complete text