When the City Teaches the Car: Label-Free 3D Perception from Infrastructure
Zhen Xu, Jinsu Yoo, Cristian Bautista, Zanming Huang, Tai-Yu Pan, Zhenzhen Liu, Katie Z Luo, Mark Campbell, Bharath Hariharan, Wei-Lun Chao 1 The Ohio State University 2 Google 3 Cornell University 4 Stanford University 5 Boston University · 2026
Read When the City Teaches the Car: Label-Free 3D Perception from Infrastructure by Zhen Xu, Jinsu Yoo, Cristian Bautista, Zanming Huang, Tai-Yu Pan, Zhenzhen Liu, Katie Z Luo, Mark Campbell, Bharath Hariharan, Wei-Lun Chao 1 The Ohio State University 2 Google 3 Cornell University 4 Stanford University 5 Boston University (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-16742.txt.
Introduction
Xu and co-authors let roadside sensors teach self-driving cars, with no labels at all. Roadside units watch fixed spots and see the same scenes repeatedly. They learn local 3D detectors from unlabeled data on their own. Their predictions stream to passing cars as pseudo-labels. The cars train a standalone detector from that supervision. At test time no roadside help is needed. In CARLA tests, the pipeline reached 82.3% AP on vehicles. That nears a fully supervised upper bound.
Catalog record
| CREATOR | Zhen Xu, Jinsu Yoo, Cristian Bautista, Zanming Huang, Tai-Yu Pan, Zhenzhen Liu, Katie Z Luo, Mark Campbell, Bharath Hariharan, Wei-Lun Chao 1 The Ohio State University 2 Google 3 Cornell University 4 Stanford University 5 Boston University |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 60087 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2603.16742
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
When the City Teaches the Car: Label-Free 3D Perception from Infrastructure Zhen Xu, Jinsu Yoo, Cristian Bautista, Zanming Huang, Tai-Yu Pan, Zhenzhen Liu, Katie Z Luo, Mark Campbell, Bharath Hariharan, Wei-Lun Chao 1 The Ohio State University 2 Google 3 Cornell University 4 Stanford University 5 Boston University Abstract Building robust 3D perception for self-driving still relies heavily on large-scale data collection and manual annotation, yet this paradigm becomes impractical as deployment expands across diverse cities and regions. Meanwhile, modern cities are increasingly instrumented with roadside units (RSUs), static sensors deployed along roads and at intersections to monitor traffic. This raises a natural question: can the city itself help train the vehicle? We propose infrastructure-taught, label-free 3D perception, a paradigm in which RSUs act as stationary, unsupervised teachers for ego vehicles. Leveraging their fixed viewpoints and repeated observations, RSUs learn local 3D detectors from unlabeled data and broadcast predictions to passing vehicles, which are aggregated as pseudo-label supervision for training a standalone ego detector. The resulting model requires no infrastructure or communication at test time. We instantiate this idea as a fully label-free three-stage pipeline and conduct a concept-and-feasibility study in a CARLA-based multi-agent environment. With CenterPoint, our pipeline achieves 82.3% AP for detecting vehicles, compared to a fully supervised ego upper bound of 94.4%. We further systematically analyze each stage, evaluate its scalability, and demonstrate complementarity with existing ego-centric label-free methods. Together, these results suggest that city infrastructure itself can potentially provide a scalable supervisory signal for autonomous vehicles, positioning infrastructure-taught learning as a promising orthogonal paradigm for reducing annotation cost in 3D perception. Figure 1 : Can city infrastructure teach vehicles to perceive? We explore a new paradigm where roadside infrastructure acts as distributed teachers, providing supervision to train ego perception models without manual annotations. 1 Introduction Developing robust perception for self-driving is inherently challenging: the task is safety-critical, and driving scenes are highly diverse. Conventional practice scales ego-view data collection and annotation to improve generalization across deployment environments. However, traffic distributions vary substantially across cities and regions, reflecting differences in road layout, infrastructure, and traffic patterns. As deployment expands to new areas, repeated collect–label–retrain cycles are necessary, and simply scaling this process becomes infeasible for addressing the structural mismatch between finite labeled data and open-world complexity. Meanwhile, modern cities are increasingly deploying roadside units (RSUs)—static sensors installed at intersections and along roads—to monitor traffic and support connected and cooperative intelligent transportation systems. For instance, transportation agencies in the United States and Europe have already begun deploying such infrastructure as part of large-scale connected-vehicle initiatives [ 47 , 10 ] . Unlike ego sensors, RSUs observe traffic from fixed vantage points with stable geometry and wide fields of view. This raises a compelling possibility: what if the city’s own sensing infrastructure ( e.g . , RSUs) could help train autonomous vehicles? More concretely, we propose turning increasingly prevalent RSUs into local perception experts that provide supervision to passing ego vehicles. In principle, this could reduce the repeated burden of manual ego-side annotation and enable geographically localized supervision wherever such infrastructure is available. At a broader scale, one can envision a city-wide collection of RSUs serving as distributed annotators , allowing vehicles to acquire supervision simply by driving through the city. At first glance, this idea may seem overly ambitious: training a large number of RSUs into reliable perception systems could itself require prohibitively expensive labeled data. In that case, the burden of annotations would merely shift from the vehicle to the city, rather than being fundamentally reduced. Our key insight is that RSUs possess a property that makes this circle breakable: stationarity . Because they repeatedly observe the same physical scene from fixed viewpoints over time, RSUs are more amenable to label-free learning based on temporal persistence, background stability, and recurring traffic patterns. Motivated by this insight, as shown in Fig. 1 , we investigate infrastructure perception as a self-learning teacher, leading to a fully label-free ecosystem for 3D perception. In this paper, we instantiate this new learning paradigm as infrastructure-taught, label-free 3D perception and conduct a concept-and-feasibility study. Because existing real-world multi-agent datasets remain limited in geographic diversity and coverage [ 33 , 41 ] and do not fully support controlled evaluation of our setting, we prepare a simulated dataset CIVET ( CI ty-as-a-teacher for ego- VE hicle T raining), collected in a multi-agent environment built on CARLA [ 8 ] and V2XVerse [ 29 ] —following a common first step in V2X research, where simulation provides a reproducible testbed for principled study [ 64 , 22 ] . In a nutshell, CIVET spans four towns, each instrumented with 12 RSUs, covering both urban and rural environments ( cf . Section 4.1 ). Two towns provide contrasting traffic characteristics, while the remaining towns are incorporated to study the scalability of the proposed paradigm. With this setup, we make the problem concrete and testable through a three-stage pipeline. In the first stage, each stationary RSU learns its own perception model from unlabeled observations by exploiting temporal consistency in dynamic traffic scenes. In the second stage, their predictions are broadcast to passing ego vehicles and aggregated as pseudo-label supervision for ego training. In the third stage, the ego detector is trained offline and operates independently at test time, carrying distilled knowledge from the city. We systematically evaluate the feasibility of this paradigm through baselines, bottleneck analysis, and upper-bound comparisons. Within a town, our fully label-free pipeline achieves 82.3% AP for detecting vehicles with CenterPoint [ 71 ] , compared to a fully supervised ego upper bound of 94.4% ( cf . Table 1 ). This observation is consistent across different detectors [ 18 , 71 ] and geolocations. We further investigate scalability and geographic generalization by jointly training across four towns using all their RSUs (48 in total), and evaluating on the combined test set with larger diversity , achieving 82.7% AP for vehicles compared to an upper bound of 91.0% ( cf . Table 5 ). Finally, we demonstrate complementarity with existing ego-centric label-free methods. Combining pseudo-labels from a representative ego-centric approach (Oyster [ 78 ] ) with our infrastructure-generated supervision yields an additional \sim 10 AP improvement over using either source alone ( cf . Table 5 ). In summary, our contributions are three-fold: • We formulate a new research direction , infrastructure-taught 3D perception , which investigates how city infrastructure can act as distributed teachers for training ego vehicle, potentially without manual supervision. • We develop a label-free pipeline in which stationary RSUs learn local detectors from unlabeled observations and provide pseudo-label supervision to passing ego vehicles, producing a standalone ego detector at test time. • We present the first systematic concept-and-feasibility study of this paradigm, including baselines, bottleneck analysis, upper bounds, and analyses of scalability and complementarity. Remark. Our setting is distinct from prior V2X and collaborative perception works [ 62 , 75 , 21 , 84 , 41 ] , which perform online multi-agent fusion at inference time and rely on labeled training data. We use infrastructure only during training: RSUs act as unsupervised, stationary teachers whose predictions are distilled into a single ego-only model. Training is label-free, inference is ego-only, and no V2X