RAVEN: Radar Adaptive Vision Encoders for Efficient Chirp-wise Object Detection and Segmentation Anuvab Sen, Mir Sayeed Mohammad, Saibal Mukhopadhyay Abstract We introduce RAVEN , a deep learning architecture for processing frequency-modulated continuous-wave (FMCW) radar data that is designed for high computational efficiency. RAVEN reduces computation by using a learnable antenna mixer module on independent receiver state space encoders (SSM) to compress the virtual MIMO array into a compact set of learned features and by performing per-chirp inference with a calibrated early-exit rule, so the model reaches a decision using only a subset of chirps in a radar frame. These design choices yield up to 170× lower computation and 4× lower end-to-end latency than conventional frame-based radar backbones, while achieving state-of-the-art detection and BEV free-space segmentation performance on automotive radar datasets. 1 Introduction Millimeter–wave radars are increasingly central to perception on autonomous ground and aerial platforms. Compared to cameras and LiDAR, they remain robust under adverse weather/lighting and directly sense relative velocity via Doppler. These qualities, combined with radar’s lower size, weight, and power profile, make it an attractive sensing modality for mobile platforms 8 ; 4 . Recent 4D imaging radars further extend range and reliability under fog and rain, offering stronger velocity tracking 41 ; 21 ; 1 . Yet higher spatial and Doppler resolution comes with a steep cost: data volume and compute scale rapidly with antenna count and bandwidth 25 ; 35 ; 2 , making them infeasible for embedded platforms and high-speed applications 23 ; 33 ; 8 . Most existing deep learning pipelines for radar perception follow a frame-based paradigm. They first collect all ADC samples for an entire radar frame, apply a sequence of fast Fourier transforms (FFTs) along range, angle, and Doppler dimensions to construct high-resolution range–angle–Doppler (RAD) tensors, and then run dense convolutional or transformer backbones on these tensors. This design exposes rich spatial–Doppler structure, but it also fixes latency to at least one frame interval and requires expensive dense processing on large 3D feature maps, which is problematic on resource-constrained platforms. Sequential models that operate directly on streaming analog-to-digital converter (ADC) signals have emerged as a promising alternative: by processing chirps as they arrive, they reduce peak memory usage and can, in principle, make decisions earlier than frame-based models 29 ( Figure 1 (a)). However, existing lightweight sequential approaches often struggle on more complex tasks such as object detection. We identify two key reasons. First, they typically compress or mix receiver channels early in the pipeline, discarding the explicit spatial localization information provided by a multiple-input multiple-output (MIMO) array. Second, in Doppler-division multiplexed (DDM) systems, they do not explicitly separate the contributions of different transmit antennas that are spectrally interleaved but remain latent in each receiver stream ( Figure 2 ). Ignoring this structure causes the virtual-array elements of different transmitters to be aliased together, which degrades angle estimation and, in turn, detection accuracy. We propose an efficient radar data processing architecture that keeps this MIMO structure explicit while remaining streaming-friendly ( Figure 1 (b)). In a MIMO radar, N_{tx} transmitter antennas and N_{rx} receiver antennas form a large “virtual array”: each receiver element views a target with a distinct phase profile determined by the array geometry, and the combination of transmitter–receiver pairs yields many virtual elements with fine angular resolution ( Figure 2 ). If the model mixes receiver channels too early or ignores which transmitter generated which echo, this virtual-array structure is lost, and recovering angle information becomes difficult, leading to degraded detection performance or the need for heavier decoders. To avoid this, we first process samples from each receiver channel independently so the encoder can learn per-antenna chirp features. We then introduce a lightweight cross-antenna attention module that learns how to combine per-chirp features across receiver channels using a small set of learnable virtual-array queries. This module effectively acts as a learnable beamformer: it reconstructs virtual-array features directly from the streaming signals without constructing range–angle–Doppler (RAD) tensors or using computationally expensive FFT based pipelines. The attention mixer adds negligible overhead to a streaming state space backbone while maintaining strong spatial localization information. We also take advantage of the evolving nature of the scene motion between radar chirps to enable early decisions in object detection. Because adjacent chirps contribute primarily differential motion (Doppler) information, detection performance saturates after a small number of chirps in a radar frame, beyond which additional chirps yield diminishing returns. We therefore train with early-chirp supervision and deploy a calibrated stopping rule that triggers as soon as the latent state of the sequence model stabilizes, reducing both encoder FLOPs and latency. Figure 1 : (a) Comparison of traditional radar processing paradigms: frame-wise CNN encoders, chirp-wise recurrent models, and sample-wise streaming SSM pipelines. (b) Our spatial-aware hybrid architecture preserves per-RX structure, performs cross-antenna attention and extracts chirp-wise virtual-array features for lightweight detection. (c) Runtime–performance characterization showing our method achieves higher accuracy at significantly lower latency and compute compared to existing radar perception models 25 ; 5 ; 29 ; 28 . Figure 2 : MIMO radar virtual antenna formation and multiplexing. (a) N_{tx} transmitters and N_{rx} receivers form N_{tx}\!\times\!N_{rx} virtual antennas. RX channels read simultaneously. (b) TDM: TX elements fire sequentially. (c) DDM: TX elements fire spectrally interleaved FMCW pulses; virtual-array information is mixed in frequency per receiver. The key contributions of this paper are: 1. Physics-inspired spatial mixing for streaming ADC: A cross-antenna attention module following independent RX processors that separates latent TX structure (DDM) to recover virtual-MIMO cues without reconstructing full RAD cubes. 2. Hybrid design for efficiency: Optimal placement of a cross-attention module between chirp-modeling channel-SSMs and frame-encoding chirp-SSM for the best compromise between computation and spatial resolving capacity. 3. Sub-frame low-latency detection: A chirp-wise SSM backbone that updates online and supports early decision via a calibrated stopping rule on chirp states, reducing encoder FLOPs and end-to-end latency ( Figure 1 (c)). 2 Related Work Classical object detection pipelines in radar vision first extract sparse point clouds (PC) via CFAR from range/angle/Doppler tensors and then run point-based or pseudo-image detectors (e.g. PointPillars variants) 27 ; 14 . While bandwidth-friendly and easy to fuse, these miss weak returns and often require multi-frame accumulation, which increases latency and can create motion ghosts. To preserve structural density, recent works process range–Doppler maps, range–angle heatmaps, or full RAD tensors with CNNs/Transformers 2 ; 25 ; 5 ; 22 . However, these 3D grids (e.g., 256\!\times\!64\!\times\!12 in a 3TX 4 RX configuration) are costly to transfer and infer on 27 . For high-antenna imaging radars, forming and processing full RAD cubes in real time is especially demanding 25 . These limitations motivate sequential ADC processing models that avoid constructing large tensors and reduce peak memory/latency. End-to-end models on time-domain ADC aim to learn task-optimal transforms beyond fixed FFTs 40 . Chirp-wise sequential encoders further reduce peak memory by updating an internal state as samples arrive; ChirpNet, for instance, reports \sim\!15\times fewer parameters than CNN baselines 29 . However, many lightweight sequential designs compress or mix receiver (RX) channels early, weakening spatial localization and angle cues latent in the MIMO array. Deep state space models (SSMs) offer linear-time streaming with long-range dependence via structured state updates 7 ; 32 ; 6 , making them natural time-series backbones; yet without explicit cross-antenna correlation, even SSM-based encoders risk discarding geometry that originates from the radar physics. Beyond radar, there is a broad literature on anytime and early-exit inference for deep models. CNNs such as MSDNet add intermediate classifiers with confidence- or entropy-based stopping to trade accuracy for compute on a per-input basis 9 . Transformer variants (DeeBERT, FastBERT) attach lightweight heads and use entropy/consistency criteria to decide when to stop 37 ; 18 . Radar-specific work has also begun to exploit temporal structure across frames for better perception 15 , but typically assumes full-frame access and dense feature maps, whereas our focus is on chirp-wise, sub-frame decisions from streaming ADC. In contrast, RAVEN combines these two: we design a radar physics inspired encoder that sequentially processes raw ADC that preserves MIMO structure via cross-antenna mixing and enables chirp-wise early decisions by applying a calibrated stopping rule on the slow-time SSM state, rather than introducing heavy auxiliary heads. 3 Methodology 3.1 Design Motivation We explicitly leverage the signal and array physics of FMCW MIMO radar when designing RAVEN’s encoder, instead of treating ADC samples as generic time-series data. In an FMCW radar, each target generates a beat frequency tied to its range and Doppler, and an N_{\mathrm{rx}} -element array encodes angle through deterministic phase shifts across antennas. These spatial phase patterns form the steering vector that enables angular resolution 31 ; 10 . Conventional sequential encoders often ignore this structure. For example, if each RX channel is reduced to a scalar and then averaged or passed through a shared 1\times 1 mix to form a per-chirp token, the operation is equivalent to applying a fixed uniform beamformer. This collapses the N_{rx} dimensional receiver array response into a single value, discarding the relative phase differences that encode angle. Downstream layers then receive tokens stripped of spatial diversity, making angle recovery significantly harder. The problem becomes worse in Doppler-division multiplexed (DDM) MIMO radars, where RX channels already contain linear mixtures of multiple TX waveforms. If the encoder does not explicitly preserve or disentangle these TX-specific components (akin to matched filters), early tokenization further mixes virtual-array responses. This additional entanglement degrades the network’s ability to learn angular structure and ultimately reduces detection accuracy. This motivates two design choices in RAVEN : • Per-RX fast-time processing: maintain separate encoders for each RX channel so that per-antenna phase and amplitude structure is preserved. • Explicit cross-antenna mixing: use a lightweight attention-based module that learns steering-like weights across RX channels and latent TX structure, instead of relying on implicit spatial learning in deep backbones. These physics-driven constraints keep the encoder compact while retaining the spatial information needed for accurate localization. Figure 3 : RAVEN Architecture: (1) Fast-time per-RX SSMs compress I/Q into compact 2-D tokens; (2) cross-antenna attention fuses RX channels and expands to virtual-MIMO features; (3) a chirp-wise SSM updates the state online across chirps; (4) a learned projection maps features to a T\times H\times W grid; (5) lightweight decoders produce detection heatmaps/boxes and segmentation. 3.2 RAVEN Model Architecture RAVEN turns streaming ADC samples into BEV detections and freespace maps through five stages ( Figure 3 ): • Fast-time SSMs: each RX channel is processed independently by a small state space model to produce a compact token per RX and chirp. • Cross-antenna attention: per-chirp RX tokens are fused with a lightweight attention module that learns spatial correlations and forms virtual MIMO features. • Chirp-wise SSM: a slow-time SSM reads the chirp sequence, maintaining a hidden state so the model can update online and support anytime inference. • Spatial projection: sequence features are mapped into a T\times H\times W grid suitable for 2D decoding. • Lightweight decoders: shallow CNN decoders produce detection heatmaps/boxes and freespace segmentation maps. Notation. Each radar frame is a sequence of N_{c} chirps (slow-time), and each chirp has N_{s} fast-time samples. The receiver (RX) has N_{\mathrm{rx}} channels and the transmitter (TX) has N_{\mathrm{tx}} channels. We use complex (I/Q) samples, so the input channel dimension is 2N_{\mathrm{rx}} . A frame is \mathbf{X}\in\mathbb{R}^{N_{c}\times N_{s}\times 2N_{\mathrm{rx}}} , with axes (slow -time, fast-time, channels). Bold capitals denote tensors, bold lower-case vectors, and plain symbols denote dimensions. We write \mathrm{LN} for LayerNorm, \sigma(\cdot) for SiLU, and \mathrm{Pool}_{K} for adaptive average pooling to length K . 3.2.1 Parallel RX-channel SSM encoders (fast-time) For receiver r\!\in\!\{1,\dots,N_{\mathrm{rx}}\} and chirp k\!\in\!\{1,\dots,N_{c}\} , let \mathbf{x}_{r,k}\in\mathbb{R}^{N_{s}\times 2} be the fast-time I/Q sequence for RX r at chirp k . Each RX uses its own state space encoder \mathrm{SSM}_{r}:\mathbb{R}^{N_{s}\times 2}\rightarrow\mathbb{R}^{N_{s}\times 2} (implemented with a Mamba block), and we compute \tilde{\mathbf{z}}_{r,k}=\mathrm{SSM}_{r}(\mathbf{x}_{r,k})\in\mathbb{R}^{N_{s}\times 2},\mathbf{f}_{r,k}=\mathrm{Pool}_{1}\!\big(\tilde{\mathbf{z}}_{r,k}^{\top}\big)\in\mathbb{R}^{2}. Stacking all receivers gives \mathbf{F}_{k}=\big[\mathbf{f}_{1,k},\dots,\mathbf{f}_{N_{\mathrm{rx}},k}\big]\in\mathbb{R}^{N_{\mathrm{rx}}\times 2},\mathbf{F}\in\mathbb{R}^{N_{c}\times N_{\mathrm{rx}}\times 2}. Intuition. Each RX stream is summarized to a tiny per-chirp token that still carries per-antenna range/phase information, providing a compact but geometry-aware input to the cross-antenna attention stage. 3.2.2 Cross-antenna attention & virtual MIMO expansion For chirp k , let \mathbf{F}_{k}\!\in\!\mathbb{R}^{N_{\mathrm{rx}}\times 2} be the per-RX summaries from the fast-time SSMs. We first expand each RX to d -dim tokens and add a learnable RX embedding: \mathbf{H}^{\mathrm{rx}}_{k}=\mathbf{W}_{\mathrm{in}}\mathbf{F}_{k}+\mathbf{E}^{\mathrm{rx}}\;\in\;\mathbb{R}^{N_{\mathrm{rx}}\times d},\mathbf{E}^{\mathrm{rx}}=[\mathbf{e}^{\mathrm{rx}}_{1},\ldots,\mathbf{e}^{\mathrm{rx}}_{N_{\mathrm{rx}}}]^{\!\top}. We introduce a bank of learnable TX queries \mathbf{Q}\!\in\!\mathbb{R}^{N_{\mathrm{tx}}\times d} that probe the RX tokens via cross-attention (queries = TX, keys/values = RX). With pre-norm, \mathbf{q}=\mathrm{LN}(\mathbf{Q}),\hskip 10.00002pt\mathbf{k}=\mathrm{LN}(\mathbf{H}^{\mathrm{rx}}_{k}),\hskip 10.00002pt\mathbf{v}=\mathbf{H}^{\mathrm{rx}}_{k}, the TX-updated tokens are \mathrm{Attn}(\mathbf{q},\mathbf{k},\mathbf{v})=\mathrm{softmax}\!\left(\frac{\mathbf{q}\mathbf{k}^{\top}}{\sqrt{d}}\right)\mathbf{v}\;\in\;\mathbb{R}^{N_{\mathrm{tx}}\times d}. We apply TX-side residual and feed-forward: \tilde{\mathbf{T}}=\mathbf{Q}+\mathrm{Attn}(\mathbf{q},\mathbf{k},\mathbf{v}) \mathbf{T}=\tilde{\mathbf{T}}+\mathrm{FFN}\!\big(\mathrm{LN}(\tilde{\mathbf{T}})\big)\;\in\;\mathbb{R}^{N_{\mathrm{tx}}\times d}. Next, for every (r,t) pair we concatenate the corresponding RX and TX tokens and project to a compact two-dimensional feature: \mathbf{p}_{r,t}=\mathbf{W}_{\mathrm{pair}}\,[\,\mathbf{h}^{\mathrm{rx}}_{r};\,\mathbf{t}_{t}\,]\;\in\;\mathbb{R}^{2},\hskip 10.00002pt\mathbf{W}_{\mathrm{pair}}\in\mathbb{R}^{2\times(2d)}. Stacking over r and t yields \mathbf{P}_{k}\in\mathbb{R}^{N_{\mathrm{rx}}\times N_{\mathrm{tx}}\times 2} , which we vectorize and normalize to form the per-chirp output: \mathbf{y}_{k}=\mathrm{LN}\!\big(\mathrm{vec}(\mathbf{P}_{k})\big)\;\in\;\mathbb{R}^{2N_{\mathrm{rx}}N_{\mathrm{tx}}}. Over all chirps this gives \mathbf{Y}=\big[\mathbf{y}_{1},\ldots,\mathbf{y}_{N_{c}}\big]\in\mathbb{R}^{N_{c}\times(2N_{\mathrm{rx}}N_{\mathrm{tx}})}. Intuition. In Figure 4 (a) the TX queries act like learnable steering vectors that search the field of RX tokens, producing TX-specific summaries \mathbf{T} . Pairwise fusion [\mathbf{h}^{\mathrm{rx}}_{r};\mathbf{t}_{t}]\mapsto\mathbb{R}^{2} then yields a compact feature for every virtual MIMO pair (r,t) , enabling the network to emphasize phase-consistent returns across antennas (DDM compatible) without constructing range–angle–Doppler (RAD) tensors. Figure 4 : (a) Attention Mixer: Learnable transmitter queries are used to extract Doppler-division multiplexed information from the receiver signal in the time domain. These are fused together to form the virtual antenna array for retrieving the MIMO information. (b) Early Decision Supervision: During training, decoders take outputs from multiple chirp levels, and loss is computed simultaneously 13 , forcing the model to converge on earlier chirps. 3.2.3 Chirp-wise (slow-time) SSM backbone We compress the channel dimension and prepare slow-time features: \mathbf{z}_{k}=\sigma\!\big(\mathbf{W}_{\mathrm{pre}}\;\sigma(\mathbf{W}_{\mathrm{red}}\mathbf{y}_{k})\big)\in\mathbb{R}^{D}, \mathbf{Z}_{\ast}=[\mathbf{z}_{1},\dots,\mathbf{z}_{N_{c}}]\in\mathbb{R}^{N_{c}\times D}, with \mathbf{W}_{\mathrm{red}}\in\mathbb{R}^{D\times(2N_{\mathrm{rx}}N_{\mathrm{tx}})} , \mathbf{W}_{\mathrm{pre}}\in\mathbb{R}^{D\times D} and an optional shallow MLP. We use Mamba-style structured SSMs that support both streaming updates and parallel training. The final slow-time representation is \mathbf{Z}_{\ast}=\text{SSM}(\mathbf{Z})\in\mathbb{R}^{N_{c}\times D}. Intuition: the SSM keeps a compact state while reading chirps in order, enabling online/anytime decisions without needing the full frame. 3.2.4 Encoder–decoder projection and heads Detection branch. We project slow-time features to a compact spatio–temporal grid and decode object heatmaps and box offsets. \displaystyle\underbrace{\mathbf{U}=\mathrm{Conv1D}_{C\!=\!D\to HW}\!\big(\mathbf{Z}_{\ast}^{\top}\big)}_{\mathbf{U}\in\mathbb{R}^{HW\times N_{c}}} \displaystyle\xrightarrow{\ \mathrm{Pool}_{T_{\mathrm{det}}}\ }\mathbf{U}_{\mathrm{det}}\in\mathbb{R}^{HW\times T_{\mathrm{det}}} \displaystyle\xrightarrow{\ \mathrm{reshape}\ }\mathbf{S}_{\mathrm{det}}\in\mathbb{R}^{T_{\mathrm{det}}\times H\times W}. A shallow Conv–LN–SiLU stack with bilinear upsampling maps \mathbf{S}_{\mathrm{det}} to a high-resolution feature map \mathbf{A} , from which 1{\times}1 heads predict classification scores and box offsets, yielding \mathrm{Det}=[\mathbf{P},\mathbf{R}] : \mathbf{P}=\mathrm{sigmoid}\!\big(\mathrm{Conv}_{\text{cls}}(\mathbf{A})\big),\hskip 10.00002pt\mathbf{R}=\mathrm{Conv}_{\text{reg}}(\mathbf{A}), Segmentation branch. We use an analogous projection to produce features for drivable-area (freespace) segmentation. An analogous projection with temporal pooling of length T_{\mathrm{seg}} produces \mathbf{S}_{\mathrm{seg}}\in\mathbb{R}^{T_{\mathrm{seg}}\times H\times W} , followed by a similar Conv–LN–SiLU + upsampling stack to obtain segmentation logits \mathbf{M}=\mathrm{Conv}\big(\mathrm{Up}(\sigma(\mathrm{LN}(\mathrm{Conv}(\mathbf{S}_{\mathrm{seg}}))))\big) 3.3 Sub-frame Decision Framework For a constant-velocity target, information about v accumulates via coherent computation across the chirps. Doppler FFT needs N_{c} chirps to achieve velocity resolution \Delta v , but detection often tolerates coarser \Delta v . This observation suggests that a sequential detector need not wait for all N_{c} chirps: it can stop once its internal state has stabilized to a sufficient prediction. 3.3.1 Training approach to enable sub-frame decision We implement this idea via multi-prefix supervision ( Figure 4 -b). Let \mathcal{L}=\{L_{1},\dots,L_{M}\}\subseteq\{1,\dots,N_{c}\} be a set of chirp-prefix lengths (with L_{M}=N_{c} ). For a frame, the encoder produces \mathbf{Z}_{\ast}\in\mathbb{R}^{N_{c}\times D} . For each L\in\mathcal{L} we take the prefix \mathbf{Z}_{\ast}^{(L)}=\mathbf{Z}_{\ast}[:,1{:}L,:]\in\mathbb{R}^{L\times D} and pass it through the same projection and decoders to obtain \widehat{\mathrm{Det}}^{(L)}\in\mathbb{R}^{3\times H^{\prime}\times W^{\prime}},\hskip 20.00003pt\widehat{\mathrm{Seg}}^{(L)}\in\mathbb{R}^{1\times H^{\prime\prime}\times W^{\prime\prime}}. \ell_{\text{det}} combines classification and box regression, and \ell_{\text{seg}} is segmentation loss. All prefixes are supervised against the same ground-truth targets (\mathrm{Det}^{\star},\mathrm{Seg}^{\star}) for the frame, yielding a deep-supervision objective \displaystyle\mathcal{L}_{\text{task}}=\sum_{L\in\mathcal{L}}\Big[\,\ell_{\text{det}}\big(\widehat{\mathrm{Det}}^{(L)},\,\mathrm{Det}^{\star}\big) \displaystyle+\ell_{\text{seg}}\big(\widehat{\mathrm{Seg}}^{(L)},\,\mathrm{Seg}^{\star}\big)\,\Big] 3.3.2 Early inference rule Let \mathbf{Z}_{\ast}^{(L)}=\{z_{1},\dots,z_{L}\}\in\mathbb{R}^{L\times D} denote the chirp-wise latent states. For each new chirp z_{L} , we measure its novelty relative to the earlier bag of chirps via the minimum cosine distance d_{L}\;=\;\min_{1\leq j