Mode-as-Sequence: Translating Multimodal Motion Prediction into Unified Sequential Mode Modeling
Zikang Zhou, Haibo Hu, Xinhong Chen Yifan Zhang, Nan Guan, Yung-Hui Li, Chun Jason Xue, Jianping Wang · 2026
Read Mode-as-Sequence: Translating Multimodal Motion Prediction into Unified Sequential Mode Modeling by Zikang Zhou, Haibo Hu, Xinhong Chen Yifan Zhang, Nan Guan, Yung-Hui Li, Chun Jason Xue, Jianping Wang (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-24037.txt.
Introduction
Zhou and seven co-authors turn motion forecasting into sequence models, lining up a set of modes so mode-to-mode links are explicit. ModeSeq decodes modes one by one, each built on earlier modes, spurring diverse guesses with trust order. Parallel ModeSeq keeps the same causal links with masked self-attention while decoding all modes at once, enabling large-K inference. With Early-Match-Take-All and a ranking regularizer, ModeSeq won the 2024 Waymo LiDAR-free motion track and Parallel ModeSeq won the 2025 Interaction Prediction Challenge.
Catalog record
| CREATOR | Zikang Zhou, Haibo Hu, Xinhong Chen Yifan Zhang, Nan Guan, Yung-Hui Li, Chun Jason Xue, Jianping Wang |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 70245 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2605.24037
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Mode-as-Sequence: Translating Multimodal Motion Prediction into Unified Sequential Mode Modeling Thanks: Corresponding author: Haibo Hu ([email protected]) Zikang Zhou, Haibo Hu, Xinhong Chen Yifan Zhang, Nan Guan, Yung-Hui Li, Chun Jason Xue, Jianping Wang Abstract Multimodal motion forecasting is inherently under-supervised: each training scene provides only one realized future, yet multiple plausible futures exist. This sparse supervision often leads to mode collapse (redundant hypotheses and insufficient mode coverage) and unreliable confidence ranking when predicting a small set of trajectories. We propose Mode-as-Sequence , a unified decoding framework that translates an unordered mode set into an ordered mode sequence and explicitly models mode-to-mode dependency. Under this framework, we develop two complementary instantiations. ModeSeq performs recurrent mode decoding, where each mode is generated conditioned on the previously generated modes, encouraging diverse, non-redundant hypotheses with calibrated confidence ordering. To remove the mode-by-mode autoregressive bottleneck, we further propose Parallel ModeSeq , which preserves the same causal dependency using masked mode-to-mode self-attention while decoding all modes in a single forward pass, enabling efficient large- K K inference and scalable joint-scene prediction. To learn representative modes and calibrated confidence under sparse labels, we introduce Early-Match-Take-All (EMTA) and its joint-scene extension MA-EMTA , together with a lightweight ranking regularizer that reduces confidence inversions. Extensive experiments on large-scale benchmarks demonstrate consistent improvements in both ranking-oriented metrics and best-of- K K accuracy across datasets, horizons, and object types. In the Waymo Open Dataset challenges, ModeSeq achieves 1st place in the 2024 LiDAR-free motion prediction track, and Parallel ModeSeq achieves 1st place in the 2025 Interaction Prediction Challenge, validating the effectiveness of Mode-as-Sequence for both accuracy and efficiency. I Introduction Handling the intrinsic uncertainty of real-world traffic is a central hurdle for autonomous driving. A major source of such uncertainty lies in multimodal agent behaviors: conditioned on the same past observation and map context, multiple plausible futures (e.g., yield vs. proceed, turn vs. go straight) may all be valid. Accurately forecasting a small set of representative trajectories with meaningful confidence scores is therefore critical for safe interaction and downstream decision making. This challenge has been actively studied on large-scale benchmarks such as the Waymo Open Motion Dataset [ 9 ] and Argoverse 2.0 [ 44 ] (as well as nuScenes [ 4 ] and INTERACTION [ 48 ] ), where predictors must jointly balance trajectory accuracy, mode diversity, and ranking quality. Fig. 1 : Illustration of three different ways to model future trajectories: (a) Parallel Mode Modeling, where all future modes are generated simultaneously without inter-mode dependency; (b) Sequential Mode Modeling, where modes are generated autoregressively, with each mode conditioned on the previously generated ones; and (c) Parallel Sequential Mode Modeling, which combines parallel generation with causal inter-mode conditioning. In (c), the visual markers indicate the causal receptive field: the second mode can access the first mode, while the third mode can access both the first and second modes. The numbers around each trajectory indicate the mode confidence. Despite extensive progress in multimodal prediction [ 5 , 31 , 19 , 34 , 15 ] , learning a calibrated multimodal distribution remains fundamentally difficult because real-world data typically provides only one ground-truth future per scene. A prevalent remedy is winner-take-all (WTA) or multiple-choice learning, where only the best-matched mode receives strong supervision while the remaining modes are weakly trained or ignored. In practice, this often leads to (i) mode collapse (near-duplicate trajectories) and (ii) ambiguous confidence learning (indistinguishable scores among overlapped modes), as also discussed in prior multimodal modeling analyses [ 27 , 3 ] . To compensate for the missing multimodal supervision, many pipelines shift to generating a large pool of candidates (dense goals or excessive anchors) and then heuristically selecting a sparse representative subset, e.g., via rule-based post-processing such as non-maximum suppression (NMS) [ 50 , 13 , 31 ] . While dense prediction can improve coverage, it introduces two practical limitations. First, post-processing requires careful hyper-parameter tuning that may not transfer across diverse scenarios. Second, accurately extracting the truly representative modes from a large candidate set is itself non-trivial, and the selection step may sacrifice trajectory quality in realistic deployment. These issues become even more pronounced for modern Transformer/graph predictors [ 53 , 37 , 29 , 52 ] , where the decoding complexity and ranking ambiguity grow with the candidate pool size. Motivated by these limitations, we seek an end-to-end solution that directly outputs a sparse set of diverse and high-quality trajectories without dense candidates or heuristic selection. We observe a shared property of mainstream multimodal decoders: they generate all K modes in one shot (in parallel), treating each mode prediction largely independently. In anchor-free designs, the distinction between modes often relies on head-specific parameters; under WTA-like training, this distinction is unstable and difficult to control, which encourages duplicated hypotheses. In anchor-based designs, diversity is partially offloaded to anchors, but selecting a sparse anchor set that adapts to each scene is difficult—hence the tendency to use many anchors and revert back to dense prediction [ 5 , 31 , 50 , 13 ] . Overall, under the parallel mode modeling paradigm, producing diverse multimodal forecasts with sparse mode prediction faces intrinsic obstacles. To address sparse multimodality at its root, we advocate a different decoding paradigm: sequential mode modeling (Mode-as-Sequence). Instead of predicting an unordered set of modes in parallel, we construct a mode sequence and decode one mode at a time, where each new mode is conditioned on previously decoded modes. This design explicitly captures mode-to-mode correlations: the model can reason about what futures have already been covered and allocate later modes to complementary behaviors, thereby improving diversity and reducing duplicates without relying on dense candidates, anchors, or post-processing. Crucially, sequential mode modeling also induces a natural ordering among modes, which can be exploited to learn better confidence ranking. In Figure 1 , we illustrate the difference between three mode prediction strategies: (a) Parallel Mode Modeling, where all future modes are generated simultaneously without dependency between them; (b) Sequential Mode Modeling (ModeSeq), where modes are generated sequentially with each mode conditioned on the previous one; and (c) Parallel Sequential Mode Modeling (Parallel ModeSeq), which combines the advantages of parallel generation with sequential mode conditioning, preserving causal dependencies between modes while ensuring efficient parallel inference. Built upon the sequential formulation, we introduce an Early-Match-Take-All (EMTA) training strategy. Intuitively, EMTA encourages the decoder to produce a mode that matches the ground-truth as early as possible in the mode sequence with high confidence. Meanwhile, once a matched mode is produced, EMTA pushes the remaining modes to vacate duplicated futures and explore other plausible behaviors, improving mode coverage with negligible degradation in trajectory accuracy. This simultaneously alleviates mode collapse and makes confidence learning easier, since earlier modes are explicitly trained to be more representative. To further enhance modeling capacity under the new paradigm, we adopt an iterative refinement decoder that stacks multiple decoding layers. Between layers, we perform mode rearrangement to align the mode order with the learned confidence ranking, coordinating the refinement process with EMTA and stabilizing the mode sequence structure. This refinement-and-reorder strategy is compatible with query-based decoders [ 52 ] and scales effectively to stronger encoders. While recurrent sequential decoding provides an elegant mechanism for mode dependency modeling,