Make Your AUV Adaptive: An Environment-Aware Reinforcement Learning Framework For Underwater Tasks

Yimian Ding, Jingzehua Xu, Guanwen Xie, Shuai Zhang, Yi Li, } @ ​ m ​ a ​ i ​ l ​ s . t ​ s ​ i ​ n ​ g ​ h ​ u ​ a . e ​ d ​ u . c ​ n , l ​ i ​ y ​ i ​ @ ​ s ​ z . t ​ s ​ i ​ n ​ g ​ h ​ u ​ a . e ​ d ​ u . c ​ n . {@mails.tsinghua.edu.cn,[email protected].} · 2025

Read Make Your AUV Adaptive: An Environment-Aware Reinforcement Learning Framework For Underwater Tasks by Yimian Ding, Jingzehua Xu, Guanwen Xie, Shuai Zhang, Yi Li, } @ ​ m ​ a ​ i ​ l ​ s . t ​ s ​ i ​ n ​ g ​ h ​ u ​ a . e ​ d ​ u . c ​ n , l ​ i ​ y ​ i ​ @ ​ s ​ z . t ​ s ​ i ​ n ​ g ​ h ​ u ​ a . e ​ d ​ u . c ​ n . {@mails.tsinghua.edu.cn,[email protected].} (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2506-15082.txt.

Introduction

Yimian Ding and co-authors present a 2025 learning framework that makes underwater robots adapt to the water around them. Such vehicles must work in flows that shift and cannot be fully predicted.

Older learning setups ignore the flow field. Their design adds a module that reads flow data live and folds it into the state the learner sees. That gives the vehicle a sharper sense of its setting and its choices. The vehicle's own shape traits also enter the tuning process. A large language model helps refine the setup in rounds, based on conditions and training results. Wide tests showed gains in performance, robustness, and adaptation.

Catalog record

CREATORYimian Ding, Jingzehua Xu, Guanwen Xie, Shuai Zhang, Yi Li, } @ ​ m ​ a ​ i ​ l ​ s . t ​ s ​ i ​ n ​ g ​ h ​ u ​ a . e ​ d ​ u . c ​ n , l ​ i ​ y ​ i ​ @ ​ s ​ z . t ​ s ​ i ​ n ​ g ​ h ​ u ​ a . e ​ d ​ u . c ​ n . {@mails.tsinghua.edu.cn,[email protected].}
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2506-15082

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Make Your AUV Adaptive: An Environment-Aware Reinforcement Learning Framework For Underwater Tasks Yimian Ding, Jingzehua Xu, Guanwen Xie, Shuai Zhang, Yi Li, } @ ​ m ​ a ​ i ​ l ​ s . t ​ s ​ i ​ n ​ g ​ h ​ u ​ a . e ​ d ​ u . c ​ n , l ​ i ​ y ​ i ​ @ ​ s ​ z . t ​ s ​ i ​ n ​ g ​ h ​ u ​ a . e ​ d ​ u . c ​ n . {@mails.tsinghua.edu.cn,[email protected].} Abstract Thisstudypresentsanovelenvironment - awarereinforcementlearning ( RL ) frameworkdesignedtoaugmenttheoperationalcapabilitiesofautonomousunderwatervehicles ( AUVs ) inunderwaterenvironments . DepartingfromtraditionalRLarchitectures , theproposedframeworkintegratesanenvironment - awarenetworkmodulethatdynamicallycapturesflowfielddata , effectivelyembeddingthiscriticalenvironmentalinformationintothestatespace . Thisintegrationfacilitatesreal - timeenvironmentaladaptation , significantlyenhancingtheAUVssituationalawarenessanddecision - makingcapabilities . Furthermore , theframeworkincorporatesAUVstructurecharacteristicsintotheoptimizationprocess , employingalargelanguagemodel ( LLM )- basediterativerefinementmechanismthatleveragesbothenvironmentalconditionsandtrainingoutcomestooptimizetaskperformance . Comprehensiveexperimentalevaluationsdemonstratetheframeworkssuperiorperformance , robustnessandadaptability . I INTRODUCTION AUVs are widely utilized in the marine domain due to their flexibility and practicality in areas such as resource exploration [ 1 ] , ecological monitoring [ 2 ] , and underwater rescue [ 3 ] . However, performing more precise tasks in complex ocean environments, such as trajectory tracking and data collection [ 4 ] , demands a more robust and adaptive AUV control method and poses higher requirements on AUV’s maneuverability [ 5 ] . In recent years, the emergence of Reinforcement Learning (RL) has somewhat addressed these issues. RL trains robots to interact extensively with environment by setting appropriate reward functions, enabling policy improvement to achieve expert-level autonomous decision-making capabilities [ 6 ] . Using RL for the control of complex robotic systems has shown promising results in various applications, including drones [ 7 , 8 , 9 ] , legged robots [ 10 , 11 , 12 ] , and also underwater vehicles [ 13 ] . However, applying RL to marine robots, such as AUVs, often results in suboptimal performance [ 14 ] . This is primarily because AUVs operate in ocean environments that vary in space and time, making it difficult for control methods and structural designs to adapt to unpredictable changes in fluid dynamics, such as randomly occurring turbulence fields and ocean currents [ 15 ] . The hydrodynamic forces experienced by AUVs differ when they operate in deep water, near the seabed, or close to dynamic objects [ 5 ] . Small changes in payload or environment during different tasks can significantly impact hydrodynamics, such as maneuverability in rotation and translation, which are difficult to model and measure [ 16 ] . When using traditional RL or classical control methods like PID controllers, this can lead to a time-consuming tuning process, where adjustments must be made based on the environment or task to support new AUV configurations [ 17 ] . Therefore, there is a need for a more general and adaptive RL training method for AUVs to support more complex field deployments. Fig. 1 : Illustration of AUV conducting underwater tasks using our proposed environment-aware RL framework . We jointly optimize the structure and control policy of the AUV through RL training integrated with its onboard environment-aware module, thereby enhancing its adaptivity in the complex ocean environment. Based on the above analysis, in this study we propose a new RL framework named environment-aware RL for underwater tasks. Our main contributions can be summarized as follows: • In addition to the original value network and policy network in RL, we add a Physics-Informed Neural Networks (PINN)-enabled environment-aware network module to capture the flow field information which is incorporated into the state space. Through iterations of RL training, enabling AUV the perceptive ability and adaptivity to the complex environment. • Subsequently, we design a large language model (LLM)-based optimization module in our framework. By providing the domain expertise and obtained RL training process with human feedback to the LLM for in-depth analysis, both the control strategy and the AUV’s structure can be adaptively optimized according to different environments and task requirements. Therefore, our proposed framework extends traditional RL approaches by introducing a joint optimization paradigm that co-advances strategy learning and structural design through environmental awareness. • Comprehensive simulations demonstrate that the proposed framework enhances expert-level decision-making capabilities for AUVs while effectively optimizing structural design, thereby significantly improving environmental adaptability. This ultimately achieves superior performance compared to traditional RL frameworks. II RELATED WORK Numerous previous studies have applied RL algorithms to train AUVs to complete underwater tasks. Tang et al. [ 18 ] implemented a Deep Q-Network (DQN) approach for dynamic AUV navigation to optimize data collection efficiency [ 19 ] . Ding et al. [ 20 ] developed a parameter prediction-based RL algorithm to address suboptimal policies convergence in multi-AUV task allocation under ocean currents. While these works demonstrate RL’s foundational potential in underwater operations, practical implementation challenges persist [ 21 ] . Under extreme disturbances, conventional RL approaches may fail to achieve theoretical optimality due to inherent limitations in AUV hydrodynamics and maneuverability [ 22 ] . Building on this foundation, several studies have enhanced RL algorithms for complex underwater environment. Sun et al . [ 23 ] improved the deep deterministic policy gradient (DDPG) algorithm by incorporating optimized sampling and motion evaluation for AUV path-following in interference-prone environments. Xu et al . [ 24 ] reformulated the underwater pursuit-evasion (UPE) game [ 25 ] as a finite-horizon Markov game and introduced a decentralized offline RL framework, achieving high performance in varying UPE scenarios. While these studies somehow improve AUV’s adaptivity in complex underwater environment, none address the AUV’s environmental adaptability or the impact of its boundary structure on training. The environment-aware RL framework proposed here effectively resolves these issues. III METHODS This section details the implementation of the proposed environment-aware RL framework. A schematic diagram of the framework’s overall structure is provided in Fig. 2. III-A The Environment-Aware Module In the proposed RL framework, the environment-aware module is responsible for introducing underwater flow field information into the RL training process, thereby enabling the AUV to perceive its surrounding environment during task execution. To better integrate this module into the RL algorithm, similar to the policy network and value network, we construct this module using a PINN [ 26 ] and name it the environment-aware network \mathcal{U}_{E}(\mathbf{\vec{x}},t;\theta) , where \theta is the network’s parameter. Specifically, the network can output the corresponding flow field velocity \mathbf{\vec{v}} and pressure p based on the input position \mathbf{\vec{x}} and time t . Fig. 2 : The overall architecture of our proposed environment-aware RL framework , which comprises of three modules: (A) Environment-aware module. (B) RL-based training module. (C) LLM-based AUV structure optimization module. In this study, we employ the dimensionless Navier-Stokes equations and the continuity equation to characterize the underwater flow field. By transforming the problem of solving the system of equations into a constrained optimization problem, the environment-aware network can be trained to simulate the fluid dynamics equations. We define the losses for the Navier-Stokes equations, the continuity equation, the boundary conditions, and the initial conditions with their weight coefficients as \alpha_{1}\mathcal{L}_{\mathcal{N}}(\mathbf{\vec{x}},t,\theta) , \alpha_{2}\mathcal{L}_{\mathcal{C}}(\mathbf{\vec{x}},t,\theta) , \alpha_{3}\mathcal{L}_{\mathcal{B}}(t,\theta) , and \alpha_{4}\mathcal{L}_{\mathcal{I}}(\mathbf{\vec{x}},\theta) , respectively. The optimization problem is formulated as follows: \begin{split}OP\colon\min_{\theta}\mathcal{L}&=\alpha_{1}\mathcal{L}_{\mathcal{N}}(\mathbf{\vec{x}},t,\theta)+\alpha_{2}\mathcal{L}_{\mathcal{C}}(\mathbf{\vec{x}},t,\theta)+\alpha_{3}\mathcal{L}_{\mathcal{B}}(t,\theta)\\ &\quad+\alpha_{4}\mathcal{L}_{\mathcal{I}}(\mathbf{\vec{x}},\theta),\end{split} (1) s.t. \mathbf{\vec{x}\in[\mathbf{\vec{x}_{min}},\mathbf{\vec{x}_{max}}]}, (2) t\in[t_{min},t_{max}], (3) where \begin{split}\mathcal{L}_{\mathcal{N}}(\mathbf{\vec{x}},t,\theta)=&(St\frac{\partial\mathcal{U}_{E}(\mathbf{\vec{x}},t;\theta)}{\partial

Read the complete text