Can Tabular Foundation Models Guide Exploration in Robot Policy Learning?

Buqing Ou, Frederike Dümbgen · 2026

Read Can Tabular Foundation Models Guide Exploration in Robot Policy Learning? by Buqing Ou, Frederike Dümbgen (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-27667.txt.

Introduction

Ou and Dümbgen offer TFM-S3, a tabular local-global hybrid for robot policy learning on a tight rollout budget. They interleave frequent local updates with rounds of global search, building a low-dimensional policy subspace by SVD for surrogate-guided refinement. A pretrained tabular foundation model scores candidates from a small context set, opening large-scale screening on a tight rollout budget. Tests on continuous control benchmarks show faster early convergence and better final performance than TD3 and population baselines under the same budget.

Catalog record

CREATORBuqing Ou, Frederike Dümbgen
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38353
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2604.27667

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Can Tabular Foundation Models Guide Exploration in Robot Policy Learning? Buqing Ou, Frederike Dümbgen Abstract Policy optimization in high-dimensional continuous control for robotics remains a challenging problem. Predominant methods are inherently local and often require extensive tuning and carefully chosen initial guesses for good performance, whereas more global and less initialization-sensitive search methods typically incur high rollout costs. We propose TFM-S3 , a tabular hybrid local-global method for improving global exploration in robot policy learning with limited rollout cost. We interleave high-frequency local updates with intermittent rounds of global search. In each search round, we construct a dynamically updated low-dimensional policy subspace via SVD and perform iterative surrogate-guided refinement within this space. A pretrained tabular foundation model predicts candidate returns from a small context set, enabling large-scale screening with limited rollout cost. Experiments on continuous control benchmarks show that TFM-S3 consistently accelerates early-stage convergence and improves final performance compared to TD3 and population-based baselines under an identical rollout budget. These results demonstrate that foundation models are a powerful new tool for creating sample-efficient policy learning methods for continuous control in robotics. I INTRODUCTION Fig. 1: Tabular Foundation Model–guided Subspace Search. The proposed method, TFM-S3, can be interleaved with any gradient-based policy training algorithm. It adds subspace-level global-search rounds as depicted in this figure. Within a dynamically updated policy parameter subspace, a set of candidate policies (colored circles) is generated. Only a small subset of candidates is evaluated through environment rollouts to form a context set (orange circles), which is used to initialize a Tabular Foundation Model. The model then predicts the performance of the remaining candidates (green circles) and ranks them by predicted return. The best candidate is evaluated and added to the context set before the next iteration. This inner-loop procedure is repeated for T_{\text{inner}} iterations, progressively guiding the search toward high-reward regions of the policy landscape with only limited rollout cost. Deep RL ( RL ) has become a prominent approach for teaching robots new skills [ 1 ] . However, as policy architectures scale to enable increasingly complex tasks, improving optimization performance remains a central challenge. A core difficulty lies in improving sample efficiency and training stability. Modern actor–critic methods, which are among the best performers in continuous control [ 2 , 3 ] , rely on neural policies with high-dimensional parameter spaces, often containing tens to hundreds of thousands of parameters [ 4 ] , resulting in highly non-convex optimization landscapes. While gradient-based methods scale to large parameter spaces, they perform inherently local updates and may converge slowly or stall in suboptimal regions [ 5 ] . To improve exploration, population-based strategies such as evolution strategies and population-based training [ 6 ] maintain diverse candidate policies, but require extensive rollout evaluations, resulting in substantial interaction cost. BO ( BO ) alleviates the need for extensive rollout evaluations by leveraging surrogate models to guide the selection of promising candidates [ 7 ] . However, classical BO becomes ineffective in high-dimensional parameter spaces due to data sparsity and the curse of dimensionality. When deep neural policies contain tens or even hundreds of thousands of parameters, constructing a reliable global surrogate from a limited number of samples becomes impractical. However, a key observation is that effective policy updates often reside in low-dimensional structured manifolds [ 8 ] . Restricting search to such subspaces reduces optimization complexity and makes surrogate modeling more feasible. At the same time, the relevant policy subspace evolves during policy training, inducing a non-stationary search geometry that violates the fixed-domain assumption underlying conventional BO [ 9 ] . This prevents the consistent aggregation of historical evaluations into a single global surrogate and calls for more flexible and adaptive surrogates. Moreover, even within a single reduced subspace, accurate return prediction remains difficult when only a small number of rollout evaluations are available. Kernel-based models such as Gaussian processes [ 10 ] require careful hyperparameter tuning and often degrade in the presence of sparse data. In contrast, recent TFM ( TFM ) [ 11 ] , pretrained with strong inductive biases [ 12 ] , provide robust small-sample generalization. When deployed within dynamically reconstructed policy subspaces, they enable reliable surrogate-guided candidate screening under limited rollout budgets, bridging gradient-based RL and sample-efficient surrogate-based search. We propose TFM-S3 (Tabular Foundation Model–guided Subspace Search), a framework that interleaves gradient-based local updates with surrogate-guided global search in a dynamically reconstructed low-dimensional policy subspace. In each search round, TFM-S3 first constructs a relevant low-dimensional policy subspace via an SVD-based procedure that captures the dominant update directions observed during recent training. Within each dynamically constructed policy subspace, TFM-S3 performs surrogate-guided global search under a limited rollout budget, as shown in Fig. 1 : a small set of candidate policies and their real returns forms the context for a TFM . The TFM serves as the surrogate and predicts returns for additional candidates, allowing for a global screening of candidates without additional rollout cost. Only the best candidate is eventually chosen for rollout and added to the context set, and the process is repeated. This process is inspired by BO with deterministic Thompson Sampling [ 13 ] , where the surrogate serves as the acquisition function to be maximized. After this subspace-level global search, the next round of local updates is initialized by the best candidate found thus far, projected back to the original parameter space. Our contributions are summarized as follows: • We address the challenge of high-dimensional policy optimization by performing search in a dynamically updated low-dimensional subspace, constructed via a gradient-informed SVD procedure that adapts to the evolving training landscape. • We tackle the difficulty of surrogate modeling in RL by leveraging a pretrained TFM which provides strong data-driven priors, enabling reliable return prediction without environment-specific tuning. • We reduce rollout requirements by combining subspace dimensionality reduction with surrogate-guided global search, requiring only a small number of real evaluations per search round while consistently accelerating convergence under identical rollout budgets. Across continuous control benchmarks, TFM-S3 consistently accelerates early-stage convergence while also improving the final policy performance averaged across seeds, suggesting that structured surrogate-guided screening enhances solution quality with higher interaction efficiency than population-based methods. II RELATED WORK II-A Zero-Order Methods in Reinforcement Learning Zero-order policy search methods optimize parameters directly through sampled rollouts without relying on gradient information. They are particularly appealing in settings where gradients are noisy or unstable. II-A 1 Population-Based and Evolutionary Methods Population-based approaches, such as evolutionary strategies and PBT ( PBT ) [ 14 ] , improve exploration by evaluating multiple policy candidates in parallel. Compared to purely local updates, they explore a broader region of the parameter space. However, these methods typically require a large number of rollouts per iteration and rely on stochastic sampling [ 15 ] . As policy dimensionality increases, maintaining adequate coverage of the search space becomes increasingly expensive due to the curse of dimensionality [ 16 ] , resulting in poor sample efficiency [ 17 ] . II-A 2 Bayesian Optimization BO reduces rollout cost by learning a surrogate model of the return function and selecting candidates via acquisition functions [ 18 ] . This predictive guidance enables more sample-efficient search. Nevertheless, in high-dimensional neural policy spaces, surrogate models become inaccurate

Read the complete text