LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations

Huangbiao Xu, Huanqi Wu, Xiao Ke, Yuxin Peng · 2026

Read LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations by Huangbiao Xu, Huanqi Wu, Xiao Ke, Yuxin Peng (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-00434.txt.

Introduction

Xu and three colleagues tackle machine learning with missing inputs. Many systems learn from video, sound, and other signals at once, but in real life one signal often drops out through a broken sensor or a privacy block. Most earlier methods assume complete data during training, which leaves them fragile when a channel goes missing. Their system, LIMSSR, reasons from whatever signals remain, using a large language model to fill gaps from context.

A masking step keeps the model honest about gaps. It weighs two lines of evidence and adjusts its confidence, so guesses built on thin context count for less. Tests on three action-quality datasets, where judges score movements, put LIMSSR ahead of the best earlier methods, even though it never saw complete training data. The result points to a leaner way of building multimodal systems that cope with gaps.

Catalog record

CREATORHuangbiao Xu, Huanqi Wu, Xiao Ke, Yuxin Peng
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE100 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.00434

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

LIMSSR: LLM-Driven Sequence-to-Score Reasoning under Training-Time Incomplete Multimodal Observations Huangbiao Xu, Huanqi Wu, Xiao Ke, Yuxin Peng Abstract Real-world multimodal learning is often hindered by missing modalities. While Incomplete Multimodal Learning (IML) has gained traction, existing methods typically rely on the unrealistic assumption of full-modal availability during training to provide reconstruction supervision or cross-modal priors. This paper tackles the more challenging setting of IML under training-time incomplete observations , which precludes reliance on a “God’s eye view” of complete data. We propose LIMSSR ( L LM-Driven I ncomplete M ultimodal S equence-to- S core R easoning), a framework that reformulates this challenge as a conditional sequence reasoning task. LIMSSR leverages the semantic reasoning capabilities of Large Language Models via Prompt-Guided Context-Aware Modality Imputation and Multidimensional Representation Fusion to infer latent semantics from available contexts without direct reconstruction. To mitigate hallucinations, we introduce a Mask-Aware Dual-Path Aggregation to dynamically calibrate inference uncertainty. Extensive experiments on three Action Quality Assessment datasets demonstrate that LIMSSR significantly outperforms state-of-the-art baselines without relying on complete training data, establishing a new paradigm for data-efficient multimodal learning. Code is available at https://github.com/XuHuangbiao/LIMSSR. 1 Introduction The success of multimodal learning in perceptual tasks largely hinges on the complementarity of multi-source information. However, a significant “Reality Gap” exists between ideal laboratory settings and real-world deployment, where missing modalities are inevitable due to sensor failures, privacy shielding, or data corruption. Consequently, Incomplete Multimodal Learning (IML) has emerged as a critical paradigm, requiring models to function robustly on arbitrary subsets of available modalities without catastrophic performance degradation. This capability is particularly vital and challenging in domains such as multimodal emotion recognition ( 55 ; 30 ) , medical analysis ( 31 ) , action recognition ( 40 ) , and action quality assessment ( 50 ) . Figure 1 : (a) Reconstruction-based and (b) Distillation/Prior-based incomplete multimodal learning methods typically rely on the unrealistic “full-modal training” assumption to generate missing data or distill priors. In contrast, (c) our LIMSSR leverages the context-aware reasoning capabilities of LLMs to infer latent semantics from available modalities under the more challenging Training-Time Incomplete Observations , eliminating the dependency on complete training data or reconstruction supervision. Current IML approaches generally fall into two categories: Reconstruction-based methods ( 57 ; 39 ) ( Figure 1 (a)), which generate missing data or latent spaces from available inputs, and Distillation/Prior-based methods ( 22 ; 55 ) ( Figure 1 (b)), which employ complete modalities as teachers to distill knowledge or learn joint priors. A common premise of these methods is the assumption of “Full-modal training,” where complete data is available during training to provide supervision or priors. This assumption, however, often contradicts reality. Due to collection costs ( 26 ) or privacy constraints ( 18 ) , training data often suffers from systemic missingness. This paper addresses this gap: learning robust representations under training-time incomplete observations without relying on a “God’s eye view” of complete modalities. The core challenge lies in the lack of a mechanism to “imagine” or complete missing semantics without direct supervision from complete data. To address this, we propose a shift in perspective, formulating incomplete multimodal learning as a conditional sequence reasoning problem, as depicted in Figure 1 (c). Given partially observed modalities and a missingness mask, the model must infer useful latent representations for the missing parts, and aggregate multimodal information for downstream prediction. We argue that Large Language Models (LLMs) are uniquely suited as the “reasoning engine” for this task. Beyond being sequence models, LLMs possess extensive world knowledge and reasoning capabilities ( 27 ) . By explicitly injecting missingness structures and task requirements via prompts, LLMs can leverage the context of available modalities and built-in priors to infer the Latent Semantics of missing parts, rather than relying on accurate pixel-level reconstruction. Furthermore, the tokenized sequence nature of LLMs allows us to model “missingness” explicitly as special tokens, converting “imputation” into a standard next-token prediction or latent representation generation task. Our goal is not to force the LLM to “understand” the specific downstream task from scratch, but to utilize its sequence reasoning and conditional mechanisms to construct a generalizable framework for missing modality imputation and fusion. To achieve this, we propose LIMSSR (short for L LM-Driven I ncomplete M ultimodal S equence-to- S core R easoning), a novel framework leveraging the sequence understanding and generative capabilities of LLMs for incomplete multimodal learning. Specifically, on the input side, we propose P rompt-Guided C ontext-Aware M odality I mputation ( PCMI ). We organize modality blocks with specific boundary tokens and represent missing modalities with distinct missing tokens to form equal-length placeholder sequences, prompting the LLM to infer hidden states based on the available context. Then, on the interface side, we introduce L LM-Driven M ultidimensional R epresentation F usion ( LMRF ). Instead of disrupting the LLM’s raw output, LMRF employ specific fusion tokens as slots to carry multidimensional fused representations, encouraging the LLM to blend multimodal information into latent representations aligned with the downstream task. To prevent sequence modeling collapse and hallucinations caused by missing data, we finally propose M ask-Aware D ual-Path A ggregation ( MDA ) on the output side. This module converts missingness into mask vectors and employs mask-aware gating to refine latent representations, dynamically balancing high-level contextual reasoning with low-level cross-modal feature aggregation. Recognizing that task loss alone is insufficient to constrain the latent space effectively, we introduce specific optimization strategies. We employ Consistency Learning to constrain the consistency between the dual paths, promoting accuracy in both representation semantics and latent missing reasoning. Furthermore, we implement Token-level Metric Regularization to force different fusion tokens to learn distinct feature dimensions, preventing feature collapse. We evaluate our approach in the context of long-term Action Quality Assessment (AQA) under incomplete multimodal settings. AQA is an emerging and challenging task with high stability requirements (e.g., sports scoring and medical evaluation) that must operate efficiently under various modality combinations. Extensive experiments demonstrate the effectiveness of our method on three public AQA datasets: FS1000 ( 43 ) , Fis-V ( 45 ) , and Rhythmic Gymnastics ( 58 ) . Our contributions are summarized as follows: • This paper novelly formulates Incomplete Multimodal AQA (IMAQA) as a conditional sequence-to-score reasoning task and systematically investigates the training-time incomplete multimodal learning where modalities are missing during both training and testing. This breaks the dependency on complete training data and better aligns with real-world data acquisition scenarios. • We propose LIMSSR, a novel LLM-driven incomplete multimodal sequence-to-score reasoning framework that employs specific tokens to guide LLMs in latent representation completion and fusion reasoning, and incorporates a mask-aware aggregation mechanism to resolve reasoning uncertainty. • We validate the missing information reasoning capability of LLMs. Our method outperforms existing supervision-dependent methods even under training-time incomplete multimodal observations, achieving the new state-of-the-art on three public benchmarks. Figure 2 : The overall framework of LIMSSR. Given incomplete data, LIMSSR first performs Prompt-Guided Context-Aware Modality Imputation, transforming available modalities into token sequences while using placeholders for missing ones. Guided by structured prompts, an LLM infers latent semantics for missing parts and consolidates multimodal information into fusion tokens. Finally, Mask-Aware Dual-Path Aggregation decodes these representations via uncertainty-calibrated reasoning and cross-modal pattern recovery paths, dynamically

Read the complete text