Gradient Free Deep Reinforcement Learning With TabPFN

David Schiff, Ofir Lindenbaum, Yonathan Efroni · 2025

Read Gradient Free Deep Reinforcement Learning With TabPFN by David Schiff, Ofir Lindenbaum, Yonathan Efroni (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-11259.txt.

Introduction

David Schiff, Ofir Lindenbaum, and Yonathan Efroni ask a sideways question in this 2025 machine learning paper. Can a model trained only for regression solve control problems? No control-task training would be involved. The model would drop straight into reinforcement learning.

Their method is called ICR-RL. It needs no extra training at all. At each step, the regression model reads past experience in its context. It then predicts good actions. Decision-making becomes a prediction problem. The authors used a recent regression foundation model. They tested on classic control benchmarks from the Gymnasium suite. The model held its own against standard methods. Those included DQN, PPO, and TRPO. Regression pre-training alone may ground future reinforcement learning systems.

Catalog record

CREATORDavid Schiff, Ofir Lindenbaum, Yonathan Efroni
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE42 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-11259

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Gradient Free Deep Reinforcement Learning With TabPFN David Schiff, Ofir Lindenbaum, Yonathan Efroni Abstract Gradient-based optimization is fundamental to most modern deep reinforcement learning (RL) algorithms; however, it introduces significant sensitivity to hyperparameters, unstable training dynamics, and high computational costs. We propose TabPFN-RL a novel gradient-free deep RL framework that re-purposes the meta-trained transformer TabPFN as a Q-function approximator. Originally developed for tabular classification, TabPFN is a transformer pre-trained on millions of synthetic datasets to perform inference on new unseen datasets via in-context learning. Given an in-context dataset of sample–label pairs and new unlabeled data, it predicts the most likely labels in a single forward pass, without gradient updates or task-specific fine-tuning. We use TabPFN to predict Q values using inference only, thereby eliminating the need for back-propagation at both training and inference. To cope with the model’s fixed context budget, we design a high-reward episode gate that retains only the top 5% of trajectories. Empirical evaluations on the Gymnasium classic-control suite demonstrate that TabPFN-RL matches or surpasses Deep Q Network on CartPole-v1, MountainCar-v0, and Acrobot-v1, without applying gradient descent or any extensive hyperparameter tuning. We discuss the theoretical aspects of how bootstrapped targets and non-stationary visitation distributions violate the independence assumptions encoded in TabPFN ’s prior, yet the model retains a surprising generalization capacity. We further formalize the intrinsic context size limit of in-context RL algorithms and propose principled truncation strategies that enable continual learning when the context is full. Our results establish prior-fitted networks such as TabPFN as a viable foundation for fast and computationally efficient RL, opening new directions for gradient-free RL with large pre-trained transformers. Introduction RL has achieved remarkable success across domains, from game playing ( Mnih et al. 2013 ) to robotics ( Haarnoja et al. 2024 ) . These advances are primarily driven by gradient-based methods, which exploit the differentiability of neural networks to optimize policies or value functions via stochastic gradient descent ( Battash, Wolf, and Lindenbaum 2024 ) . While effective at enabling end-to-end learning and long-term credit assignment, such methods are often sample-inefficient and sensitive to hyperparameters, posing challenges in sparse-reward and offline settings. In particular, RL algorithms typically require careful tuning of numerous hyperparameters—including optimizer choice, learning rate, and network architecture—which can significantly impact performance ( Adkins, Bowling, and White 2024 ) . Even different runs of the same algorithm with varying hyperparameters may yield drastically different results, necessitating extensive search procedures and substantial computational resources. Popular deep RL algorithms, such as Deep Q-Networks (DQN) ( Mnih et al. 2013 ) and Proximal Policy Optimization (PPO) ( Schulman et al. 2017b ) , exemplify these challenges. DQN, which approximates the Q-function using a neural network and relies on temporal difference updates, is particularly susceptible to instability due to the combination of bootstrapping, function approximation, and off-policy learning. This often necessitates the introduction of additional mechanisms, such as target networks and experience replay buffers, to stabilize training, which in turn introduces new hyperparameters and design choices. On the other hand, PPO, a widely used policy gradient method, improves stability through clipped surrogate objectives but remains fundamentally on-policy, which limits sample efficiency. Furthermore, both DQN and PPO assume that the policy or value function is differentiable with respect to its parameters, making them inapplicable in domains involving non-differentiable components or discrete structures. These caveats, including sensitivity to noisy gradients, reliance on differentiability, and the complexity of stabilizing training dynamics, motivate the exploration of gradient-free RL algorithms. By avoiding gradient-based optimization altogether, such methods offer the potential for greater robustness, especially in sparse, non-smooth, or highly stochastic environments. Recent progress in large language models (LLMs) has demonstrated that transformers can perform a wide range of tasks through in-context learning (ICL) ( Garg et al. 2023 ) , eliminating the need for gradient-based fine-tuning. In natural language processing, models like GPT-4 demonstrate the ability to solve diverse problems purely by conditioning on task demonstrations in the input sequence. The ability to learn in context without any gradient-based training could also be implemented for RL. Instead of training task-specific RL agents from scratch via gradient descent, we explore the conceptual possibility of a foundation model for RL that can perform RL through ICL, using past transitions to predict optimal behavior. Our work explores this direction through the use of TabPFN a transformer trained for ICL over tabular classification/regression problems, which is repurposed here as a Q-function approximator that requires no gradient-based updates during interaction. This opens the door to a novel framework for gradient-free deep RL, where policy improvement occurs purely through ICL. Several approaches have explored RL without gradient-based updates. Evolution Strategies have demonstrated competitive performance on benchmarks like Atari, albeit at high computational cost due to population-based optimization ( Salimans et al. 2017 ) . Other methods, such as KNN-based Q-learning, replace neural networks with instance-based estimators but rely on problematic assumptions about state similarity ( Martín H., de Lope, and Maravall 2009 ) . More recently, meta-RL and ICL have emerged as powerful alternatives to traditional RL. Works like RL 2 ( Duan et al. 2016 ) and Algorithm Distillation ( Laskin, Srinivas, and Abbeel 2022 ) utilize recurrent or transformer-based models to learn RL algorithms that adapt solely via context, without requiring parameter updates at test time. However, the success of both models has been evaluated mainly on discrete state spaces or a simple visual maze environment. OmniRL ( Wang et al. 2025 ) enhances this approach by training a large transformer on a variety of procedurally generated tasks, which facilitates broader generalization across RL settings. However, OmniRL has only demonstrated success in discrete state spaces. To our knowledge, all ICL RL methods do not address the issue of continual learning when the context is full. This is a problem for all ICL RL methods, as their ability to learn is limited by the size of their contexts. TabPFN-RL is a supervised, meta-trained Transformer that performs ICL for small tabular datasets and infers the most likely labels using inference only. Given a context of input–output pairs \mathcal{D}_{\text{train}} , it directly outputs the distribution over the possible labels q_{\theta}(y\mid x_{\text{test}},\mathcal{D}_{\text{train}}) for a test point, without requiring any gradient-based updates at test time. Trained on millions of synthetic supervised tasks sampled from a data-generating prior, TabPFN effectively learns a meta-algorithm capable of inferring labels or target values from small, structured datasets in a single forward pass. In this study, we propose a new approach to gradient-free RL by utilizing TabPFN as an ICL RL model. Specifically, we study whether TabPFN ’s strong inductive bias, acquired through pre-training on millions of synthetic tabular datasets, can be used for RL without relying on gradient-based updates. This method paves the way for faster, data-efficient RL. Our results demonstrate that TabPFN-RL can be effectively applied to RL and is competitive with widely used RL methods, such as DQN ( Mnih et al. 2013 ) , in Gymnasium’s classic control environments ( Towers et al. 2024 ) . Our contributions are as follows: (1) We develop a framework that uses TabPFN as a Q(s,a) function approximator in online RL. (2) We empirically evaluate our method on standard Gymnasium environments, demonstrating its effectiveness in RL. (3) We discuss

Read the complete text