Multimodal Signal Restoration with Signed Twofold Graph Learning Publication type: Original Paper

Haruki Yokota, Hiroshi Higashi, Yuichi Tanaka · 2026

Read Multimodal Signal Restoration with Signed Twofold Graph Learning Publication type: Original Paper by Haruki Yokota, Hiroshi Higashi, Yuichi Tanaka (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2605-26760.txt.

Introduction

Yokota, Higashi, and Tanaka learn signed twofold graphs and restore multimodal signals in one joint procedure. Older twofold-graph methods assume the graphs are known or force edge weights to stay non-negative, missing negative links between modalities. Their method estimates both graphs from noisy, incomplete data, recovers signed modality structure from a kernel matrix, and unrolls the solvers into a trainable network. Tests on synthetic and real sensor data beat existing baselines across noise levels.

Catalog record

CREATORHaruki Yokota, Hiroshi Higashi, Yuichi Tanaka
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE55465
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2605.26760

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Multimodal Signal Restoration with Signed Twofold Graph Learning Publication type: Original Paper Haruki Yokota, Hiroshi Higashi, Yuichi Tanaka Abstract Multimodal signals on sensor networks are commonly modeled under the twofold graph assumption (TGA), which represents spatial structure and inter-modality relations as two separate graphs. Existing TGA-based signal restoration methods, however, either assume the graphs are known or restrict edge weights to be non-negative, preventing them from capturing negative inter-modal correlations. We address both limitations by formulating joint signal restoration and twofold graph learning as MAP estimation under a matrix normal prior, where the spatial and modality graph Laplacians appear directly as precision matrices. The resulting non-convex objective is solved by alternating minimization: The signal is updated via conjugate gradient applied to the arising Sylvester-type linear system; the graphs are updated via primal-dual hybrid gradient (PDHG). We further propose a method to estimate the signed structure of the modality graph from the dominant eigenspace of a complementary kernel matrix, which is then used in PDHG to update edge magnitudes. These iterative solvers are then unrolled into a feedforward network, with regularization weights and step sizes as layer-wise trainable parameters. Experiments on synthetic multimodal graph signals and a real Japan meteorological dataset confirm that the proposed method outperforms existing baselines across a range of noise levels and missing-data patterns. keywords Graph Learning, Graph Signal Processing, Multimodal Signals, Algorithm Unrolling 1]The University of Osaka, Suita, Japan 2]Kansai University, Suita, Japan \issuevolumeyear 2026 \issuevolumenumber xx \issueelocation e \articledatabox Received xx xxxxx 2026; Revised xx xxxxx 2026 ISSN xxxx-xxxx; DOI 10.1561/xxx.xxxxxxxx © H. Yokota, H. Higashi and Y. Tanaka \OAstatement This is an Open Access article, distributed under the terms of the Creative Commons Attribution licence ( http://creativecommons.org/licenses/by-nc/4.0/ ), which permits unrestricted re-use, distribution, and reproduction in any medium, for non-commercial use, provided the original work is properly cited. \creditline *Corresponding author: [email protected]. The preliminary version of this work was presented at the Asia Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC) 2025 Yokota et al. 2025 . 1 Introduction In sensor networks, sensors often capture a rich variety of data simultaneously, like video/audio streams and temperature/pressure readings. Such data are called multimodal signals. In practice, real-world multimodal signals are imperfect, suffering from noise or missing values. Signal restoration for those multimodal signals, including denoising and interpolation, is therefore crucial for a wide range of applications, such as autonomous driving, human-computer interaction, and medical diagnostics Xiao et al. 2022 ; Liu et al. 2024 ; Liu et al. 2024a ; Arceo et al. 2025 . A key principle in signal restoration on sensor networks is utilizing underlying structures of data, whose structures are mathematically represented as graphs Shuman et al. 2013 ; Ortega et al. 2018 ; Tanaka et al. 2020 . Multimodal signals on a network can be modeled as a spatial graph, with underlying spatial structure, and inter-modality relations are represented by a modality graph. This assumption is known as twofold graph assumption (TGA) Nagahama & Tanaka 2022 . Previous works for signal restoration with TGA face two key challenges. First, many approaches either assume graphs are known a priori or rely on heavily parameterized networks to learn them Shahid et al. 2016 ; Wang et al. 2020 ; Zheng et al. 2022 , limiting applicability in data-scarce scenarios. Second, all existing TGA methods are designed for unsigned graphs, modeling only pairwise similarities Kojima et al. 2026 . This neglects negative correlations across modalities—for example, in environmental monitoring, the anti-correlation between temperature and precipitation is naturally encoded as a negative edge weight. Such a relationship can be represented by signed graphs Dinesh et al. 2025 ; Yokota et al. 2025a , but unsigned graphs cannot. In this paper, we address these limitations by proposing a method that: 1) formulates joint signal restoration and twofold graph learning as MAP estimation under a matrix normal prior, linking the graph Laplacians directly to the signal prior; 2) learns a spatial unsigned graph and a modality signed graph simultaneously, using a sign-magnitude decoupling with a spectral sign estimator; and 3) unrolls the resulting alternating iterative solvers into an interpretable deep network whose hyperparameters are trained end-to-end. Specifically, we model the multimodal signal as a matrix normal random variable Gupta & Nagar 2000 with graph Laplacians as the precision matrices, yielding a MAP objective that couples data fidelity with twofold graph smoothness. We solve the resulting non-convex problem by alternating minimization over three variables: the signal is updated by solving a Sylvester-type linear system via conjugate gradient (CG); the spatial graph is updated by primal-dual hybrid gradient (PDHG) Condat 2013 ; and the signed modality graph is updated by decoupling sign from magnitude, estimating the sign structure via subspace iteration and solving for edge magnitudes with PDHG. Finally, we unroll the CG and PDHG iterations into a feedforward network Monga et al. 2021 , with regularization weights and step sizes as layer-wise trainable parameters. We evaluate the proposed method on synthetic multimodal graph signals and a real Japan meteorological dataset. Experimental results demonstrate that the proposed method outperforms competing baselines in both denoising and interpolation across a range of noise levels and missing-data patterns. Notation: We use bold uppercase letters for matrices ( \mathbf{A} ), and bold lowercase letters for vectors ( \mathbf{a} ). The corresponding non-bold letters with subscripts denote their elements ( A_{ij} and a_{i} ). The j th column of matrix \mathbf{A} is denoted as the vector \mathbf{a}_{j} . The identity matrix is \mathbf{I} , and vectors of all ones and zeros are \mathbf{1}=[1,\dots,1]^{\top} and \mathbf{0}=[0,\dots,0]^{\top} , respectively. The \ell - p norm is \|\cdot\|_{p} and the Frobenius norm is denoted as \|\cdot\|_{F} . An undirected graph is denoted as \mathcal{G} , and we denote the set of N nodes in \mathcal{G} as \mathcal{V}=\{v_{1},v_{2},\dots,v_{N}\} . The topology and edge weights of \mathcal{G} are defined by a symmetric weighted adjacency matrix \mathbf{W}\in\mathbb{R}^{N\times N} , where W_{ij} represents the weight of the edge between node v_{i} and v_{j} . Throughout the paper, we assume W_{ii}=0 , i.e., the graphs have no self-loops. The degree matrix is the diagonal matrix denoted as \mathbf{D}=\operatorname{diag}(d_{1},\dots,d_{N}) , where d_{i}=\sum_{j}W_{ij} . The combinatorial graph Laplacian \mathbf{L} for unsigned graphs is given by \mathbf{L}=\mathbf{D}-\mathbf{W} , and those for signed graphs are defined as \bar{\mathbf{L}}=\bar{\mathbf{D}}-\bar{\mathbf{W}} , where \bar{\mathbf{D}} is an absolute degree matrix whose i th diagonal element is defined as \bar{d}_{ii}=\sum_{j}|\bar{W}_{ij}| . 2 Related Works In this section, we review the three bodies of work most directly related to the proposed method: multimodal graph signal restoration, unsigned graph learning, and signed graph learning. 2.1 Multimodal Graph Signal Restoration Under the twofold graph assumption (TGA) Nagahama & Tanaka 2022 ; Kojima et al. 2026 , a multimodal signal \mathbf{X}\in\mathbb{R}^{N\times M} is assumed to be smooth on a spatial graph \mathcal{G}_{s} over N nodes and on a modality graph \mathcal{G}_{m} over M modalities simultaneously. Signal restoration under TGA is then formulated as minimizing a regularizer involving both graph Laplacians. The conventional TGA-based methods only consider the sum of two graph regularizers Nagahama & Tanaka 2022 ; Kojima et al. 2026 ; \min_{\mathbf{X}}\|\mathbf{Y}-\mathcal{A}(\mathbf{X})\|_{F}^{2}+\alpha\text{tr}(\mathbf{X}^{\top}\mathbf{L}_{s}\mathbf{X})+\beta\text{tr}(\mathbf{X}\mathbf{L}_{m}\mathbf{X}^{\top}), (1) where \mathcal{A}(\cdot) is a degradation operator, and \alpha and \beta are regularization parameters. This formulation does not explicitly

Read the complete text