Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving
Jason Gibson, Anoushka Alavilli, Erica Tevere, Evangelos A. Theodorou, Patrick Spieler · 2024
Read Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving by Jason Gibson, Anoushka Alavilli, Erica Tevere, Evangelos A. Theodorou, Patrick Spieler (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00581.txt.
Introduction
Self-driving across rough ground at speed fails when the vehicle model ignores what the terrain does to traction. Jason Gibson, Anoushka Alavilli, Erica Tevere, Evangelos A Theodorou, and Patrick Spieler tie the model to camera views in a 2024 paper. A pretrained vision model reads fine detail from the scene ahead. Those features feed a hybrid dynamics model that shifts with the ground. A compressor turns the heavy features into a light runtime map.
Training and tests span hundreds of kilometers of hard off-road driving in many places. The data came from a defense research program on resilient autonomy. Slip, grip, and rolling drag are the effects the model must foresee. Planning at the edge of control is the goal.
Catalog record
| CREATOR | Jason Gibson, Anoushka Alavilli, Erica Tevere, Evangelos A. Theodorou, Patrick Spieler |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 33 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-00581
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Dynamics Modeling using Visual Terrain Features for High-Speed Autonomous Off-Road Driving Jason Gibson, Anoushka Alavilli, Erica Tevere, Evangelos A. Theodorou, Patrick Spieler Abstract Rapid autonomous traversal of unstructured terrain is essential for scenarios such as disaster response, search and rescue, or planetary exploration. As a vehicle navigates at the limit of its capabilities over extreme terrain, its dynamics can change suddenly and dramatically. For example, high-speed and varying terrain can affect parameters such as traction, tire slip, and rolling resistance. To achieve effective planning in such environments, it is crucial to have a dynamics model that can accurately anticipate these conditions. In this work, we present a hybrid model that predicts the changing dynamics induced by the terrain as a function of visual inputs. We leverage a pre-trained visual foundation model (VFM) DINOv2, which provides rich features that encode fine-grained semantic information. To use this dynamics model for planning, we propose an end-to-end training architecture for a projection distance independent feature encoder that compresses the information from the VFM, enabling the creation of a lightweight map of the environment at runtime. We validate our architecture on an extensive dataset (hundreds of kilometers of aggressive off-road driving) collected across multiple locations as part of the DARPA Robotic Autonomy in Complex Environments with Resiliency (RACER) program. https://www.youtube.com/watch?v=dycTXxEosMk I Introduction In off-road autonomous driving, varying terrain geometries and properties influence the terradynamics experienced by the vehicle. Terrain geometries include obstacles, hills, trenches, and slopes, while terrain properties include stiffness, friction, and other properties that vary with the vegetation, gravel, or sand on the vehicle’s path. When driving without prior knowledge of the environment, real-time information about the terrain can provide insight into the safety and efficiency of traversing various paths. For example, Mars Science Laboratory (the Curiosity rover) experienced unanticipated wheel damage, prompting the choice of path plans involving more benign terrain [ 17 ] . Similarly, the vehicle slip experienced by the Mars Exploration Rovers (Spirit and Opportunity) was difficult to predict given the highly varied terrain [ 14 ] . Fig. 1: Architecture of dynamics learning with visual features. A feature encoder is trained end-to-end with the dynamics model on a dataset of high-dimensional visual features from a VFM. This feature encoder reduces the visual information to a low-dimensional dynamics relevant feature space. At runtime, it processes features in image space (dashed line) before projection and accumulation in a 3D map. This makes the map aggregation step computationally tractable. The 3D map is flattened to a top down 2D terrain feature map used by the dynamics model in the MPC planner. Dynamics modeling for field robots using visual inputs has been an area of research since before visual foundation models (VFMs) were available. For example, in [ 1 ] , different vehicle slip models were determined for a small number of different terrain conditions (such as grass, gravel, etc.). Though discretizing terrain types could suffice in some environments and when the robot is not operating at its hardware limits, a continuous representation of terrain is more desirable for our purposes, where terrain varies drastically and the driving is highly aggressive. VFMs are large-scale, pre-trained neural networks designed to handle a wide range of vision-related tasks, such as image classification, segmentation, and feature extraction. VFMs such as DINOv2 [ 16 ] are trained to produce general-purpose features, which could be used for applications such as classification or segmentation. Features can carry semantic meaning; for example, some feature maps may encode depth information and provide linear image segmentation. In this work, we leverage DINOv2 to extract compact and continuous representations of terrain, capturing visual details that can inform the vehicle’s dynamics. Understanding the vehicle’s surrounding terrain is especially important as the coupling of high speeds and varying ground terrain introduce complex, nonlinear, and time-varying properties in the dynamics of the vehicle as aspects such as the traction, cornering stiffness, and rolling resistance of the vehicle change. For example, the vehicle’s dynamics when driving on slippery grass differ from those on dry trails. Examples of terrain where the vehicles dynamics may be affected as a result of the terradynamics of the environment are shown in Fig. 2 . Fig. 2: Terrain geometries and properties vary significantly across the environments. Images show a selection of diverse terrain (from top left to bottom right: packed sand, muddy ditches and ruts, loose dirt trail, tall grass, dense overgrown vegetation, steep slopes) for which visual inputs of the terrain inform the changing dynamics of the vehicle. These challenges highlight the importance of integrating perceptual inputs into our dynamics model. Most importantly, the dependence on visual features will allow for the anticipation of changes to vehicle dynamics, ultimately reducing the need for model correction and adaptation as the vehicle drives on varying terrain. To this end, the contributions of this work are as follows: 1). A hybrid dynamics model (i.e., a model with both physics-based parametric and neural-network components) that predicts changes in terradynamics as a function of terrain using visual features derived from a VFM. 2). A training architecture that enables end-to-end learning of a feature encoder that can be used to lower the computational burden of tracking high-dimensional features in map space at runtime. 3). A novel method for compressing a visual feature space that is robust to projection distance and occlusion, enabling generalization to real driving scenarios. 4). Validation of our method on a large dataset of aggressive off-road driving across a variety of rugged terrains. II Related Work In off-road autonomy literature, many studies aim to improve the quality of the traversability map by incorporating visual information about the surrounding terrain [ 19 ] , [ 2 ] , [ 5 ] , [ 6 ] , [ 15 ] , [ 8 ] . For instance, [ 19 ] combines semantic labels and geometric hazard identification to determine the costs in the traversability map. [ 2 ] learns a traction model to estimate the slip parameters of the terrain and factors this into the cost of the traversability map. Some works, such as [ 8 ] , classify terrain into a discrete set of options, like smooth and rough regolith, limiting their approach. The main goal of these methods is to avoid unsafe paths. Less explored is the use of visual terrain features to inform vehicle dynamics modeling, which is necessary for accurate control and planning at a higher resolution. Other works, such as [ 12 ] and [ 10 ] , also propose hybrid models (i.e., a model that has parametric and learned components) for high-speed vehicle dynamics modeling. In both works, the vehicle drives on-road, and therefore neither approach uses visual terrain features to inform the model. Most relevant to this research are [ 13 ] and [ 3 ] . Both methods use the DINO or DINOv2 VFM to inform a model about the traversed terrain. [ 3 ] employs the visual features to inform two physical parameters, stiffness and friction, while [ 13 ] uses this visual terrain-feature-based model for adaptive control. Compared to [ 3 ] , our method outputs a richer representation of the terrain by training a feature encoder end-to-end with a dynamics model. In contrast to