Learning from Less: SINDy Surrogates in RL

Aniket Dixit, Muhammad Ibrahim Khan, Faizan Ahmed, James Brusey · 2025

Read Learning from Less: SINDy Surrogates in RL by Aniket Dixit, Muhammad Ibrahim Khan, Faizan Ahmed, James Brusey (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-18113.txt.

Introduction

Aniket Dixit and co-authors build cheap stand-in worlds for training learning agents, in work from 2025. Reinforcement learning agents learn by trial and error, which can take vast numbers of steps in the real task. A stand-in world that mimics the real one can cut that cost.

The team used an algorithm that finds sparse equations for how a system changes over time. With very few real trials, their stand-ins matched the true dynamics of two classic test tasks closely. Agents trained inside the stand-ins needed far fewer steps than agents trained in the real tasks. Their final scores were about the same. The method gives accurate, readable stand-ins, which suits labs that need cheaper and clearer training setups.

Catalog record

CREATORAniket Dixit, Muhammad Ibrahim Khan, Faizan Ahmed, James Brusey
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE16 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-18113

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Learning from Less: SINDy Surrogates in RL Aniket Dixit, Muhammad Ibrahim Khan, Faizan Ahmed, James Brusey Abstract This paper introduces an approach for developing surrogate environments in reinforcement learning (RL) using the Sparse Identification of Nonlinear Dynamics (SINDy) algorithm. We demonstrate the effectiveness of our approach through extensive experiments in OpenAI Gym environments, particularly Mountain Car and Lunar Lander. Our results show that SINDy-based surrogate models can accurately capture the underlying dynamics of these environments while reducing computational costs by 20 - 35 % 2035\text{\,}\mathrm{\%} . With only 75 75 interactions for Mountain Car and 1000 1000 for Lunar Lander, we achieve state-wise correlations exceeding 0.997 0.997 , with mean squared errors as low as 3.11 × 10 − 06 3.11\text{\times}{10}^{-06} for Mountain Car velocity and 1.42 × 10 − 06 1.42\text{\times}{10}^{-06} for LunarLander position. RL agents trained in these surrogate environments require fewer total steps ( 65 075 65\,075 vs. 100 000 100\,000 for Mountain Car and 801 000 801\,000 vs. 1 000 000 1\,000\,000 for Lunar Lander) while achieving comparable performance to those trained in the original environments, exhibiting similar convergence patterns and final performance metrics. This work contributes to the field of model-based RL by providing an efficient method for generating accurate, interpretable surrogate environments. 1 Introduction RL has revolutionized the field of artificial intelligence by enabling autonomous agents to learn optimal behavior through interaction with their environment. Despite its remarkable success in various domains, from game playing to robotics, the practical implementation of RL faces a significant challenge: the requirement for extensive environmental interactions during training. This limitation not only makes the training process computationally intensive but also poses safety risks in real-world applications where trial-and-error learning may be impractical or dangerous. To address these challenges, we propose an approach leveraging SINDy ( Brunton et al. 2016 ) algorithm to create efficient surrogate environments. Our methodology significantly reduces the data requirements while maintaining high fidelity to the original environment dynamics. We demonstrate the effectiveness of our approach using two widely-studied OpenAI Gym environments: Mountain Car and Lunar Lander, which represent distinct classes of control problems with varying degrees of complexity. Our research advances RL through several key contributions. We develop a framework for creating SINDy-based surrogate environments that accurately capture the dynamics of standard RL benchmarks while requiring minimal training data. Through rigorous empirical analysis, we demonstrate that agents trained in our surrogate environments achieve performance comparable to those trained in original environments, with a significant reduction in computational resources. We comprehensively investigate the fundamental trade-offs between model complexity and prediction accuracy in surrogate environment design, providing practical guidelines for implementation. This work represents the first application of SINDy to RL environment modeling, establishing a new paradigm for sample-efficient surrogate model development. In addition, we develop a systematic evaluation framework for quantifying the fidelity of the surrogate environment, including metrics for both state-transition accuracy and policy transfer success. The widespread adoption of RL in practical applications is currently hindered by several critical challenges. These include prohibitive computational requirements for environment simulation, particularly in complex domains requiring high-fidelity physics engines; extended training durations that can span days or weeks, limiting rapid prototyping and experimentation; substantial resource consumption in terms of processing power and energy, raising concerns about environmental impact; and safety considerations in real-world applications where direct environment interaction during training could lead to hazardous situations. 2 Methodology 2.1 Data Collection We leverage pre-trained RL models to collect high-quality state transition data with minimal sampling. Specifically, a pre-trained Soft Actor-Critic (SAC) ( Haarnoja et al. 2018 ) agent was utilized to collect data from the Mountain Car environment, with only 75 state transitions captured from a single episode. Similarly, a pre-trained Proximal Policy Optimization (PPO) ( Schulman et al. 2017 ) agent was used to gather 1000 state transitions from a single episode in the more complex Lunar Lander environment. To balance between exploration and exploitation, we implemented an \epsilon -greedy policy ( \epsilon=0.2 ) for both agents, ensuring 80% of actions followed the trained policy while 20% were randomly sampled. This strategy provided diverse state-action pairs covering both optimal paths and exploratory regions of the state space ( Sutton & Barto 2018 ) .For each transition, we recorded the current state, action taken, and resulting next state (s_{t},a_{t},s_{t+1}) , then normalized and stored the data in CSV format. 2.2 SINDy Model Development SINDy was employed to uncover the underlying dynamics of both environments from collected state transitions, generating sparse, interpretable governing equations that succinctly capture the essential behavior of the system. The model development followed a systematic four-stage process: 1. Initial Model Construction : We initially fitted basic models using a simplified feature library, ultimately culminating in the application of the Sequential Thresholded Least Squares (STLSQ) method. 2. Residual Analysis & Refinement : Prediction errors guided the progressive addition of nonlinear terms (trigonometric for Mountain Car, polynomial for Lunar Lander). 3. Parameter Optimization : Grid search across threshold and regularization values, optimizing for both sparsity and prediction accuracy. 4. Cross-Validation : Models were validated using held-out data to ensure generalization performance, with final selection based on minimal MSE. This process generated parsimonious models that effectively captured environment dynamics while excluding spurious terms through rigorous optimization. 2.3 SINDy driven model The SINDy model is subsequently integrated into a surrogate OpenAI Gym compatible environment ( Arora et al. 2022 ; Zolman et al. 2024 ) . In this framework, the original physics engine is replaced by the SINDy-based model. The new surrogate environment preserves all the key characteristics of the original implementations. The primary modification lies in the state transition function, where physics-based calculations are supplanted by trained SINDy models: s_{t+1}=f_{\text{SINDy}}(s_{t},a_{t}) (1) This approach retains the essential dynamics while significantly reducing computational overhead. The SD-RL framework facilitates efficient policy learning by leveraging a data-driven surrogate environment. Employing SINDy to model system dynamics ensures environment interpretability and computational efficiency. Input: Number of episodes N , SINDy parameters \theta_{\text{SINDy}} , RL parameters \theta_{\text{RL}} Output: Trained RL policy \pi_{\theta} 1 // Data Collection for i=1 to N do 2 Initialize environment and reset state s_{0} 3 for t=1 to max timesteps do 4 Select action a_{t} using \varepsilon -greedy policy 5 Execute a_{t} and observe next state s_{t+1} and reward r_{t} 6 Store transition (s_{t},a_{t},s_{t+1},r_{t}) in dataset \mathcal{D} 7 end for 8 end for 9 10 // Train Sparse Dynamics Model (SINDy) Train SINDy model using dataset \mathcal{D} and hyperparameters \theta_{\text{SINDy}} 11 12 // Train RL Agent in Surrogate Environment Train RL agent using surrogate environment based on learned SINDy dynamics 13 Optimize policy \pi_{\theta} using \theta_{\text{RL}} 14 15 return Trained policy \pi_{\theta} Algorithm 1 Sparse Dynamics-Driven Reinforcement Learning (SD-RL) 2.4 Experimental Setup To validate our surrogate environments, we conducted experiments comparing the performance of RL agents trained on both the original and surrogate environments. We used state-of-the-art RL algorithms appropriate for each environment’s characteristics, following the Sparse Dynamics-Driven RL approach outlined in Algorithm 1 . 3 Results 3.1 SINDy Model Performance and Policy Learning Our evaluation focused on two key aspects: (1) the accuracy of learned dynamics models and (2) the similarity of policies trained in surrogate vs. original environments. Table 1: SINDy Model Validation Metrics and Library Function Impact

Read the complete text