Scaling-Aware Data Selection for End-to-End Autonomous Driving Systems Tolga Dimlioglu 1, Nadine Chang 2, Maying Shen 2, Rafid Mahmood 2,3, Jose M. Alvarez 2 Abstract Large-scale deep learning models for physical AI applications depend on diverse training data collection efforts. These models and correspondingly, the training data, must address different evaluation criteria necessary for the models to be deployable in real-world environments. Data selection policies can guide the development of the training set, but current frameworks do not account for the ambiguity in how data points affect different metrics. In this work, we propose Mixture Optimization via Scaling-Aware Iterative Collection (MOSAIC), a general data selection framework that operates by: (i) partitioning the dataset into domains; (ii) fitting neural scaling laws from each data domain to the evaluation metrics; and (iii) optimizing a data mixture by iteratively adding data from domains that maximize the change in metrics. We apply MOSAIC to autonomous driving (AD), where an End-to-End (E2E) planner model is evaluated on the Extended Predictive Driver Model Score (EPDMS), an aggregate of driving rule compliance metrics. Here, MOSAIC outperforms a diverse set of baselines on EPDMS with up to 80% less data. 1 Introduction Large-scale deep learning models are fueled by diverse data collection efforts ( 41 ; 35 ) . This practice is particularly prominent in physical artificial intelligence (AI) applications such as autonomous driving (AD), where video clips are collected over different locations, weather, and traffic conditions ( 18 ; 12 ) . It is computationally inefficient to train models on all collected data, which in physical AI can scale to hundreds of millions of hours of clips. This necessitates data mixture selection policies to construct and grow training sets of diverse and influential samples that maximize desired performance metrics. Figure 1 : (a-b) The data pool is partitioned into a set of discrete domains which may each contribute to performance improvement of different evaluation tasks at varying rates. (c-d) Example application in autonomous driving: two clusters representing different driving contexts—Pittsburgh (curvy suburban roads) and Las Vegas (dense urban traffic). Data from separate contexts influence different rule-compliance metrics at distinct rates. Dataset selection and optimization has been broadly studied from various perspectives. For instance, influence (or duplicate) estimation techniques use feature information to select useful data samples 1 ; 5 ; 49 , while active learning strategies optimize over this feature space ( 46 ; 38 ) . Large language models (LLMs) and their multi-modal extensions have successfully leveraged scaling laws to forecast how model performance improves with dataset size ( 26 ; 21 ; 58 ) ; this premise has expanded to other applications including AD ( 3 ) . Further, as data collection becomes increasingly complex, scaling laws are used to determine optimal mixtures of data from explicit domains (e.g., different languages, math, coding) ( 23 ; 56 ; 36 ) . Although these methods present a general opportunity for physical AI systems, they are not immediately usable for applications that require both understanding and interacting with diverse real-world scenarios for three main reasons. First, physical AI systems are evaluated over a set of potentially competing metrics 14 ; 57 . Second, different data samples can influence different combinations of these metrics at various rates. Finally, the data pool is not necessarily immediately separable into subsets that have consistent, predictable influence on the metrics. Existing data mixture methods assume well-defined and homogeneous domains. However, they overlook the heterogeneous and metric-dependent improvement rates that arise when data sources influence different aspects of performance at varying rates 20 ; 54 . For example, a physical AI system such as an autonomous vehicle must progress along a route, follow driving rules, and avoid collisions ( 14 ) . High-traffic and pedestrian-heavy driving clips, when used for training, may impact certain metrics more than others. Moreover, finding such a subset of potential training data that has shared effects on the metrics requires careful selection and mining. In this work, we develop a data selection and mixture optimization policy that addresses the present physical AI challenges of multiple competing metrics and imprecise data partitions. To address the challenge of imprecise data domains, we first partition a data pool into a set of separable clusters, within which we can rank samples on their influence to the metrics. We estimate the impact of each cluster on each metric, and correspondingly, an overall utility function that aggregates all the metrics. This impact is measured in terms of scaling laws that estimate the improvement to the metrics if more data from a specific cluster were used for training. Finally, we iteratively add new data to the training set by identifying the cluster which will maximize the expected gain to the aggregate utility with each additional data point. In this way, we optimize the mixture of data from our generated partitions. Figure 1 summarizes the challenges. We apply our framework, Mixture Optimization via Scaling-Aware Iterative Collection ( MOSAIC ), for End-to-End (E2E) autonomous driving, where the challenges of data heterogeneity and metric competition are particularly pronounced. The goal is to optimize the Extended Predictive Driving Model Score (EPDMS), which aggregates a diverse set of rule-compliance metrics. MOSAIC is more data efficient than existing methods and achieves better EPDMS performance than naïve baselines with up to 82% less additional data. Our contributions are: • We propose MOSAIC , a generic data mixture optimization pipeline that (i) clusters and ranks data, (ii) models domain-specific data scaling, and (iii) mines samples to maximize the expected gain over aggregate metrics. • We apply MOSAIC to End-to-End Autonomous Driving (E2E AD) on the NAVSIM and OpenScene benchmarks using the challenge winning Hydra-MDP model 33 , where it achieves substantially higher driving performance than existing data selection and mixture baselines, and improve data efficiency by up to 82%. Moreover, MOSAIC achieves the full training performance while requiring 42% less data samples. • We empirically demonstrate the necessity and robustness of our joint clustering and scaling procedure. First, MOSAIC outperforms baselines regardless of the clustering approach (e.g., semantic captions, geolocation). Second, embedding scaling laws on top of clustering significantly outperforms clustering-only strategies. This underscores the importance of our principled data selection strategy, which leverages the estimated improvement rates of different data clusters to maximize model performance under limited data budgets. 2 Related Works Data Mixtures. Recent work has highlighted the importance of how data from different domains are combined for large-scale model training 39 ; 53 ; 17 ; 56 ; 36 ; 55 ; 54 ; 23 . DoReMi 53 employs two proxy models to estimate domain weights based on excess loss, which are later used to reweigh domains when training a larger model. DOGE 17 , tracks domain-specific gradients while training the proxy model to better capture inter-domain dynamics. Chameleon 54 instead leverages kernel similarity scores computed in the model’s latent space to assign adaptive weights to data from different sources. Another line of work treats data mixture optimization as a regression problem: many small proxy models are trained with varying mixtures, and a regressor is then fit to predict the optimal mixture at larger scale 36 ; 56 . A particularly relevant approach to ours is ADO 23 , which begins with a random data mixture and fits scaling estimators on the fly during training. The gradients of these estimators are used for mixture reweighting. However, ADO does not model how performance scales with different data sources in isolation, and it requires a temporal averaging mechanism with multiple hyperparameters to maintain the precision of scaling fits. Although the aforementioned data mixture methods assign weights to samples from different domains, these weights can also be interpreted as sampling probabilities for constructing mixtures with varying domain ratios. In our experiments, we adopt Chameleon 54 as a baseline, since it has been shown to outperform other mixture algorithms. Data Pruning & Selection. Data pruning aims to identify a compact subset of training data by removing redundant samples while preserving model performance 47 . In vision tasks, 1 proposed removing visually similar samples using cosine similarity in the CLIP 43 feature space. Follow-up works extended this idea to specialized domains such as object detection 25 and fairness-aware multimodal learning 48 . It has also been shown analytically that optimal pruning strategies can improve power-law scaling behavior 49 . A closely related line of work, Active learning (AL), aims to maximize model performance improvement under a limited annotation budget 44 ; 59 . In this setting, the model has access to a large unlabeled data pool and, based on some selection signal the most informative samples are identified to be used for training. Early works focused on using the model’s prediction uncertainty, quantified through posterior probabilities 32 , classifier margins 45 , or entropy 24 . A notable method, CoreSet 46 , seeks representation diversity by mining samples that maximize coverage in the latent space. Other data selection strategies quantify the sample importance using expensive signals such as influence on model updates 37 , gradient-based criteria 8 or forgetting score 50 . End-to-End Autonomous Driving. This task aims to train planner models that map raw sensory inputs directly to control commands. Early approaches 4 ; 10 ; 42 learned control actions from RGB inputs via imitation learning, while later works incorporated richer input modalities such as LiDAR and navigational commands 9 ; 52 . Recently, conventional open-loop metrics have been shown to correlate poorly with closed-loop driving quality 34 ; 13 . This motivated the development of simulation benchmarks that better reflect real-world driving performance 14 and, AD models to employ probabilistic and rule-compliant trajectory planners 22 ; 7 ; 33 . Figure 2 : Overview of the proposed MOSAIC framework. (a) The pool \mathcal{D}_{pool} is clustered and ranked by sample importance. (b) Cluster-wise scaling laws are fitted on pilot runs to estimate how performance scales with added data. (c) Samples are then iteratively mined from the cluster with the highest estimated marginal gain under the fitted scaling laws. 3 MOSAIC 3.1 Main Problem We want to train a Deep Neural Network (DNN) f(\cdot;\mathcal{D}) on a dataset \mathcal{D} to perform a given task. We evaluate model performance using a set of R metrics \mathcal{G}_{r}(f(\cdot;\mathcal{D}),\mathcal{D}_{val}) for r\in\{1,\cdots,R\} , where \mathcal{D}_{val} is a held-out validation dataset. For brevity, we denote \mathcal{G}r(f(\cdot;\mathcal{D}),\mathcal{D}_{val}) by \mathcal{G}_{r}(\mathcal{D}) . To balance the trade-offs between the metrics, we use a utility function U(\{\mathcal{G}_{r}(\mathcal{D})\}_{r=1}^{R}) that aggregates each metric into a final score. We do not assume about the structure of the utility function; for example, the simplest approach would be a summation U(\cdot)=\sum_{r=1}^{R}\mathcal{G}_{r}(\mathcal{D}) . We initialize with a current training dataset \mathcal{D}_{train} and a data pool \mathcal{D}_{pool} . Given a budget B , our goal is to select a subset \mathcal{D}_{sel}\subset\mathcal{D}_{pool} with |\mathcal{D}_{sel}|=B that maximizes the improvement in model performance when f is retrained on the combined dataset \mathcal{D}_{train}\cup\mathcal{D}_{sel} . Formally, we write \max_{\begin{subarray}{c}\mathcal{D}_{sel}\subset\mathcal{D}_{pool}\\ |\mathcal{D}_{sel}|=B\end{subarray}}U\Big(\{\mathcal{G}_{r}(\mathcal{D}_{train}\cup\mathcal{D}_{sel})\}_{r=1}^{R}\Big) (1) To solve problem ( 1 ), we must determine how each data sample added in \mathcal{D}_{sel} influences each of the metrics, while optimizing the trade-offs between these metrics to maximize U(\cdot) . For instance, in our AD application, f(\cdot;\mathcal{D}) is a planner model that maps sensory inputs to a predicted driving trajectory. Here, our goal is to identify driving clips that optimize the Extended Predictive Driving Model Score (EPDMS) 27 ; 6 , which is an aggregate of R=9 closed-loop rule compliance scores: NC, DAC, DDC, TLC, EP, TTC, LK, HC, EC. Each driving clip can showcase only certain aspects of driving rule compliance, e.g. driving on a curvy road might improve lane keeping while degrading the comfort, however our goal is to elevate the model performance across all metrics. Consequently, solving this problem requires disentangling the relationships between the data samples and the metrics before optimizing the trade-offs between them. Algorithm 1 Mixture Optimization via Scaling-Aware Iterative Selection (MOSAIC) 1: Pool dataset \mathcal{D}_{pool} , number of clusters M , sample selection budget B . 2: Selected dataset \mathcal{D}_{sel} 3: \{\mathcal{D}_{pool}^{i,ranked}\}_{i=1}^{M}=\texttt{ClusterAndRank}(\mathcal{D}_{pool},M) 4: \{\Delta\hat{U_{i}}(n)\}_{i=1}^{M}=\texttt{GetScalings}(\{\mathcal{D}_{pool}^{i,ranked}\}_{i=1}^{M}) 5: \mathcal{D}_{sel}\leftarrow\{\} 6: b_{i}\leftarrow 0 for all i\in\{1,\dots,M\} 7: while |\mathcal{D}_{sel}|