Towards Foundational Models for Single-Chip Radar

Tianshu Huang Akarsh Prabhakara Chuhan Chen Jay Karhade, Deva Ramanan Matthew O’Toole Anthony Rowe · 2025

Read Towards Foundational Models for Single-Chip Radar by Tianshu Huang Akarsh Prabhakara Chuhan Chen Jay Karhade, Deva Ramanan Matthew O’Toole Anthony Rowe (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2509-12482.txt.

Introduction

Millimeter-wave radar is cheap and tough. It is blind to darkness and fog. But its angular resolution is poor. That flaw is worst in the single-chip units used in cars and rooms. Tianshu Huang and six co-authors aim to give radar a foundation model in 2025. They train a general transformer on raw radar data, at unusual scale.

Their dataset holds about a million samples. The samples span twenty-nine hours. The model is the Generalizable Radar Transformer. It predicts three-dimensional occupancy and semantic labels. The quality is one once reserved for far sharper sensors. Raw data beats the usual compressed forms by a wide margin. That margin is worth ten times the data. Accuracy also grows steadily as data scales.

Catalog record

CREATORTianshu Huang Akarsh Prabhakara Chuhan Chen Jay Karhade, Deva Ramanan Matthew O’Toole Anthony Rowe
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE70 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2509-12482

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Foundational Models for Single-Chip Radar Tianshu Huang Akarsh Prabhakara Chuhan Chen Jay Karhade, Deva Ramanan Matthew O’Toole Anthony Rowe Abstract mmWave radars are compact, inexpensive, and durable sensors that are robust to occlusions and work regardless of environmental conditions, such as weather and darkness. However, this comes at the cost of poor angular resolution, especially for inexpensive single-chip radars, which are typically used in automotive and indoor sensing applications. Although many have proposed learning-based methods to mitigate this weakness, no standardized foundational models or large datasets for the mmWave radar have emerged, and practitioners have largely trained task-specific models from scratch using relatively small datasets. In this paper, we collect (to our knowledge) the largest available raw radar dataset with 1M samples (29 hours) and train a foundational model for 4D single-chip radar, which can predict 3D occupancy and semantic segmentation with quality that is typically only possible with much higher resolution sensors. We demonstrate that our Generalizable Radar Transformer (GRT) generalizes across diverse settings, can be fine-tuned for different tasks, and shows logarithmic data scaling of 20% per 10 × 10\times data. We also run extensive ablations on common design decisions, and find that using raw radar data significantly outperforms widely-used lossy representations, equivalent to a 10 × 10\times increase in training data. Finally, we roughly estimate that ≈ \approx 100M samples (3000 hours) of data are required to fully exploit the potential of GRT. 1 Introduction As a compact, inexpensive [ 27 ] , and robust solid-state sensor, mmWave radars are ideal for sensing applications ranging from simple automatic door openers [ 58 ] to autonomous drones [ 11 ] or vehicles [ 53 , 62 ] . mmWave radars are rich sensors which can directly measure range and velocity while capturing a unique range of material properties [ 18 ] ; however, this comes at the cost of poor angular resolution typically on the order of 15^{\circ} – orders of magnitude worse than cameras or lidars [ 59 ] . Radar data are typically processed into radar point clouds (Fig. 1 ) derived using Constant False Alarm Rate (CFAR) peak detectors [ 36 , 52 ] combined with Angle-of-Arrival estimation techniques [ 60 ] . However, this is a substantially lossy process: while raw radar data suffers from unique noise patterns such as “bleed” and side lobes [ 25 ] , weak reflectors and other signals can be hidden in this noise, which would ordinarily be filtered out. On the other hand, raw spectrum (4D range-Doppler-azimuth-elevation data cubes [ 50 ] ) can be unintuitive and difficult to interpret compared to lidar point clouds or camera images, and include properties such as specularity and Doppler which lack straight-forward Cartesian interpretations [ 18 ] . As such, many machine learning methods have been proposed [ 69 , 24 , 45 ] to exploit 4D radar data from single-chip radars, achieving remarkable performance on 2D scene understanding tasks. However, due to the dominance of CFAR point clouds in radar processing, as well as the high data rate of raw mmWave radar data, most radar toolchains only process point-cloud data. Tooling for raw I/Q (in-phase/quadrature) data is often brittle, poorly documented, and largely unsupported by radar vendors, severely limiting the availability and scale of both raw mmWave datasets and the models which operate on raw data. To rectify this limitation, we develop an open-source toolchain and associated large dataset specifically for 4D mmWave radar data. Training a radar-to-lidar model and fine-tuning for a range of other tasks, we demonstrate the surprising effectiveness of mmWave radar models trained at scale (Fig. ). Going further, just as large foundational models [ 4 ] have greatly accelerated the pace of innovation in computer vision and natural language processing, we believe that a foundational model for raw mmWave radar trained at even larger scale could similarly supercharge the advancement of radar sensing techniques. Contributions In this paper, we develop a full stack 1 1 1 Our data collection system, dataset, code, and model can be found via our project site: https://wiselabcmu.github.io/grt/ . for collecting data, training, and evaluating a transformer for 4D single-chip radar to quantify both the potential costs and benefits of training a foundational model at scale. To summarize our contributions: 1. We develop a compact, lightweight multimodal data collection system (Sec. 3.1 ) capable of collecting synchronized raw radar, Lidar, and camera data which can be operated as a handheld device. Our system can be easily replicated using off-the-shelf components, 3D printed parts, and our open-source software. 2. Using this data collection system, we collect a dataset, I/Q-1M ( One Million IQ Frames ), consisting of 29 hours of data – 8 \times longer than the next largest publicly available raw radar dataset – split between indoor, outdoor-handheld, and bike-mounted settings, each with different radar configurations (Sec. 3.2 ). 3. Finally, using our dataset, we train a Generalizable Radar Transformer (GRT), which can output depth maps and segmentation images with quality which is typically only possible with much higher resolution radars. Using GRT, we then run ablations on common design choices (Sec. 5.1 ), quantify the scalability of GRTs with increasing dataset and model size (Sec. 5.2 , 5.4 ), and demonstrate that our GRT can be readily fine-tuned for other tasks and settings (Sec. 5.3 ), including obtaining state-of-the-art performance on the Coloradar [ 23 ] dataset with 30-minutes of fine-tuning. Figure 1 : While a transformer can generate Lidar-like depth and bird’s eye view images, traditional CFAR point clouds are noisy, and have poor angular resolution – especially in the elevation axis. Key Findings We summarize our key findings as follows: • Radar models can generalize to different settings and radar configurations (Sec. 5.2 ), as well as across objectives (with some fine-tuning). This suggests great potential for a cross-domain foundational model to improve and accelerate the development of new radar models. • Using raw data yields outsized performance gains , equivalent to more than a 10\times increase in training data (Sec. 5.1 ). While existing datasets largely focus on CFAR point clouds or other processed representations, we believe that more emphasis should be placed on making raw data available for research. • Existing mmWave radar datasets are vastly undersized . 24 hours of training data is not enough to saturate even a 4M parameter model! Our analysis suggests that at least 100 \times more data is required to exploit the full potential of radar transformer models (Sec. 5.4 ). Limitations Despite its size compared to previous datasets, I/Q-1M is quite small compared to the datasets used to train modern vision transformers, which can exceed billions of samples [ 68 ] . I/Q-1M also only includes daylight conditions and fair weather, and lacks the scale to capture “edge cases” that would be represented in a larger dataset [ 22 ] . Finally, since I/Q-1M uses a single type of radar, we cannot evaluate generalization across different antenna configurations — only radar configurations. 2 Related Work Figure 2 : Our data collection rig in its handheld (right) and bike-mounted (left) configurations; see App. A for additional images. 4D Solid State Radar Excluding mechanical radars, which perceive the world as

Read the complete text