Revisiting Birds Eye View Perception Models with Frozen Foundation Models: DINOv2 and Metric3Dv2
Seamie Hayes, Ganesh Sistu, Ciaran Eising · 2025
Read Revisiting Birds Eye View Perception Models with Frozen Foundation Models: DINOv2 and Metric3Dv2 by Seamie Hayes, Ganesh Sistu, Ciaran Eising (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-08118.txt.
Introduction
Seamie Hayes, Ganesh Sistu, and Ciarán Eising test whether frozen foundation models can strengthen bird's-eye view perception for cars, in a two thousand twenty-five paper. Bird's-eye view models fuse camera images into a top-down scene map. They usually need large amounts of driving data to generalize.
The authors modified two architectures, Lift-Splat-Shoot and Simple-BEV. Into the first they plugged frozen DINOv2 features and Metric3Dv2 depth estimates. That version beat its baseline by a wide margin while using half the training data. For the second, they turned Metric3Dv2 depth into a pseudo-LiDAR point cloud replacing real LiDAR. The camera-only model gained several points of overlap accuracy. Careful reuse of big pretrained models pays off.
Catalog record
| CREATOR | Seamie Hayes, Ganesh Sistu, Ciaran Eising |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 25 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-08118
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Revisiting Birds Eye View Perception Models with Frozen Foundation Models: DINOv2 and Metric3Dv2 Seamie Hayes, Ganesh Sistu, Ciaran Eising Abstract Birds Eye View perception models require extensive data to perform and generalize effectively. While traditional datasets often provide abundant driving scenes from diverse locations, this is not always the case. It is crucial to maximize the utility of the available training data. With the advent of large foundation models such as DINOv2 and Metric3Dv2, a pertinent question arises: can these models be integrated into existing model architectures to not only reduce the required training data but surpass the performance of current models? We choose two model architectures in the vehicle segmentation domain to alter: Lift-Splat-Shoot, and Simple-BEV. For Lift-Splat-Shoot, we explore the implementation of frozen DINOv2 for feature extraction and Metric3Dv2 for depth estimation, where we greatly exceed the baseline results by 7.4 IoU while utilizing only half the training data and iterations. Furthermore, we introduce an innovative application of Metric3Dv2’s depth information as a PseudoLiDAR point cloud incorporated into the Simple-BEV architecture, replacing traditional LiDAR. This integration results in a +3 IoU improvement compared to the Camera-only model. Introduction The recent revolution in machine learning has paved the way for the creation of numerous vehicle segmentation networks for use in autonomous vehicles, all competing to achieve optimal performance. The prompt yet accurate detection of vehicles in an image is no easy task, and the utilization of large convolutional networks, or transformer models has been extensively investigated. To train these networks effectively, they require dense information from which to learn, typically derived from three sensor modalities—camera, LiDAR, and radar—used individually or in combination. These modalities are complementary: cameras deliver rich 2D colour images that enhance vehicle segmentation and lane detection. LiDAR and radar generate point clouds, significantly improving vehicle segmentation and depth perception. When integrated with camera data, they outperform models that rely solely on camera input [ 1 ] . Training autonomous perception models can be time-consuming due to the extensive data requirements and the size of datasets like nuScenes [ 2 ] , which includes 700 driving scenes each with 40 annotated keyframes. The Lift-Splat-Shoot (LSS) model, previously considered state-of-the-art, is trained for approximately 40 epochs on this dataset and achieves a performance of approximately 33 IoU. Reducing training time while maintaining performance would be remarkable, and this would require a readily available model pre-trained on a massive dataset to replace a particular task in the architecture: a foundation model. A foundation model is trained on a large, diverse dataset, for the purpose of generalisation and then applicable to tasks such as image object segmentation, depth perception, text classification, etc. The advancement of foundation models cannot be downplayed. For instance, EfficientSAM [ 3 ] processes images significantly faster than the original SAM [ 4 ] , with minimal performance loss. Likewise, for depth perception, Depth Pro [ 5 ] outperforms Metric3Dv2 [ 6 ] in monocular depth estimation, producing a 2.25-megapixel depth map in just 0.3s. We chose two autonomous bird’s eye view vehicle segmentation models for modification: Lift-Splat-Shoot (LSS) and Simple-BEV. LSS, a well-established model, shows strong performance in vehicle segmentation. In contrast, Simple-BEV, which utilizes LiDAR and radar data from nuScenes, outperforms LSS. For LSS, we implemented foundational models: DINOv2 for feature extraction [ 7 ] and Metric3Dv2 for depth estimation, using them in their frozen state to avoid fine-tuning and preserve original weights. It’s crucial to note that DINOv2 is designed to capture all image features, not just vehicles. For Simple-BEV, we experimented with Metric3Dv2 depth images as a PseudoLiDAR point cloud, aiming to replace traditional LiDAR or radar data. Results demonstrate that these foundational models significantly enhance initial performance due to their versatility. By using only camera-based solutions, which are significantly cheaper than LiDAR, there’s potential for substantial cost savings in equipment manufacturing. The main contributions of this paper are as follows: 1. The implementation of foundation models in Lift-Splat-Shoot significantly enhances results, even with only half the training data and fewer iterations. Additionally, an ablation study examines how the size of foundation models affects performance. 2. The unique implementation of metric depth images to simulate LiDAR in Simple-BEV which yields better results than the Camera-only model. Background Lift-Splat-Shoot Figure 1: Architecture Flow Diagram: Modified Lift-Splat architecture with implementation of DINOv2 and Metric3Dv2 for image feature extraction and depth estimation The LSS model consists of three primary components. The first, lifting, transforms each image into a frustum of features using a geometry transformation and an EfficientNet backbone for acquiring image features and depth estimation [ 8 ] . Next, the splat step encodes these frustums into a BEV voxel grid using PointPillars, which is then processed by a BEV encoder to produce the BEV vehicle segmentation output. The final component, shoot, involves trajectory planning for the ego vehicle, selecting the path with the lowest cost determined by a network. This paper focuses primarily on the Lift component, which pertains to the acquisition of camera features and depth estimation. Here, replace the output of the EfficientNet encoder with DINOv2 image features and Metric3Dv2 depth to facilitate increased model performance, which will be described later in Section Alteration to the Model Architecture . This modified model architecture is illustrated in Figure 1 Simple BEV Figure 2: Architecture Flow Diagram: Simple BEV model for Camera+PseudoLiDAR. The Camera-only model excludes any point clouds, while Radar and LiDAR would substitute for Pseudo LiDAR The Simple BEV architecture allows us to implement radar or LiDAR data along with camera information which makes it a very versatile model. We extract features from each image and then perform bilinear sampling to lift these features into a 3D voxel. For LiDAR and radar data, we convert the point cloud information into an occupancy grid representation. We then concatenate the two voxels together to yield a unified voxel, compress it to the BEV space, and input it to the BEV encoder, producing our BEV vehicle segmentation output. A comprehensive flow diagram is provided in Figure 2 . Additionally, we have the option to train three models: Camera-only, Camera+Radar, or Camera+LiDAR. LiDAR and radar provide the model with important depth information and also permit the model to segment vehicles not seen by the camera images due to the range of these sensors. Both modalities provide notable performance increases over the single-modal Camera-only model. DINOv2 The significance of foundation models in computer vision cannot be overstated, especially with the advent of DINOv2 [ 7 ] . Trained on over 140 million images from diverse datasets, this self-supervised model leverages a Vision Transformer-based backbone to excel in zero-shot inference. By dividing images into 14\times 14 patches, DINOv2 effectively captures long-range dependencies, allowing even its frozen version to outperform fine-tuned conventional models, as detailed in Section Results: Comparison with Original Model . The model scales feature vector length from 384 in smaller versions to larger dimensions in bigger variants. While fine-tuning DINOv2 for specific tasks is feasible, it requires significant computational resources. In our study, we employ the frozen model to sidestep these complexities, incorporating a single convolutional layer to downsample DINOv2 feature vectors to 64, thus ensuring compatibility with the EfficientNet BEV encoder. As shown in Table