Capsule Network–Based Multimodal Fusion for Mortgage Risk Assessment from Unstructured Data Sources

Mahsa Tavakoli, Rohitash Chandra, Cristián Bravo · 2025

Read Capsule Network–Based Multimodal Fusion for Mortgage Risk Assessment from Unstructured Data Sources by Mahsa Tavakoli, Rohitash Chandra, Cristián Bravo (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2510-22987.txt.

Introduction

Mahsa Tavakoli, Rohitash Chandra, and Cristian Bravo propose judging mortgage risk from free public data instead of costly private records. Standard scoring leans on structured financial data that is often secret and expensive. Their framework instead fuses text, images, and sentiment scores from open sources.

First they pick the best model for each data type, using BERT for text and VGG for images, plus a small network for sentiment. A capsule network then merges those signals into a credit score meant to mimic commercial scorecards. The approach could widen fair access to risk checks where records are scarce.

Catalog record

CREATORMahsa Tavakoli, Rohitash Chandra, Cristián Bravo
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE76 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2510-22987

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Capsule Network–Based Multimodal Fusion for Mortgage Risk Assessment from Unstructured Data Sources Mahsa Tavakoli, Rohitash Chandra, Cristián Bravo Abstract Mortgage risk assessment traditionally relies on structured financial data, which is often proprietary, confidential, and costly. In this study, we propose a novel multimodal deep learning framework that uses cost-free, publicly available, unstructured data sources, including textual information, images, and sentiment scores, to generate credit scores that approximate commercial scorecards. Our framework adopts a two-phase approach. In the unimodal phase, we identify the best-performing models for each modality, i.e. BERT for text, VGG for image data, and a multilayer perceptron for sentiment-based features. In the fusion phase, we introduce the capsule-based fusion network (FusionCapsNet) , a novel fusion strategy inspired by capsule networks, but fundamentally redesigned for multimodal integration. Unlike standard capsule networks, our method adapts a specific mechanism in capsule networks to each modality and restructures the fusion process to preserve spatial, contextual, and modality-specific information. It also enables adaptive weighting so that stronger modalities dominate without ignoring complementary signals. Our framework incorporates sentiment analysis across distinct news categories to capture borrower and market dynamics and employs GradCAM-based visualizations as an interpretability tool. These components are designed features of the framework, while our results later demonstrate that they effectively enrich contextual understanding and highlight the influential factors driving mortgage risk predictions. Our results show that our multimodal FusionCapsNet framework not only exceeds individual unimodal models but also outperforms benchmark fusion strategies such as addition, concatenation, and cross attention in terms of AUC, partial AUC, and F1 score, demonstrating clear gains in both predictive accuracy and interpretability for mortgage risk assessment. 1 Introduction Exploring the factors that influence mortgage defaults has significance in finance and risk management [ 1 ] . For example, investing in the mortgage market during periods of high defaults can lead to substantial drops in asset values, potentially leading to wealth erosion. Furthermore, high default rates can strain financial institutions [ 110 ] , impacting their lending capabilities and overall financial stability. High default rates can lead to tighter credit markets, making it harder for individuals and businesses to secure loans, further affecting economic growth and stability. Understanding these risks is crucial for investors, policymakers, and financial institutions to mitigate possible negative impacts on both individual and broader economic levels. A large body of prior research has focused on the prediction of mortgage defaults using structured data sets [ 1 ] , particularly loan characteristics and the demographics of the borrower [ 110 ] . These studies consistently show that such variables strongly influence lenders’ decisions when assessing credit risk [ 9 ] . However, small businesses, independent financial advisors, investors, and the public may also need to understand the risks of mortgage default using the factors available to them. This knowledge is essential for shaping investment strategies, supporting effective portfolio risk management, and guiding decisions related to the real estate market. Access to structured financial data is often challenging for these groups because such data is typically controlled by large institutions and restricted due to privacy concerns. This limitation reduces public access and creates a significant barrier to obtaining critical information. Publicly available data is highly valuable, as it can provide insights into mortgage risk without relying on proprietary financial datasets. Leveraging such data allows market participants with limited resources to make more informed mortgage investment decisions and enables policymakers to design evidence-based legislative or regulatory measures. In the field of data analytics, structured data has long played a key role in supporting predictive modeling tasks [ 3 ] . However, the potential of unstructured data from multimedia sources, such as text and images, remains largely untapped and can provide valuable and often unexpected insights [ 4 ] . With the rapid growth of big data [ 47 ] , the volume of unstructured and semi-structured data continues to increase, drawing significant attention from both researchers and practitioners. Unstructured data encompasses a wide range of formats, including text documents, emails, social media posts, audio files, images, videos, and other multimedia content that lack a predefined structure. In the realm of mortgage default prediction, structured data has been extensively studied [ 48 ] , while the integration of unstructured data remains in its infancy due to challenges such as data preprocessing, feature extraction, and model interpretability. Despite these hurdles, the incorporation of unstructured data represents a promising frontier for research, offering the potential to uncover novel insights, enrich feature representations, and enhance predictive accuracy [ 2 ] . Unstructured data sources that are openly accessible and readily available can include materials such as news articles and Remote Sensing images. These sources offer valuable analytical content, providing opportunities to extract meaningful insights [ 50 ] . The Earth Engine Data Catalog 1 1 1 https://developers.google.com/earth-engine/datasets/catalog hosts LiDAR (Light Detection and Ranging) data as part of its openly accessible collections through Google Earth Engine 2 2 2 https://earthengine.google.com/ . Google Earth Engine is a cloud-based platform designed by Google for analyzing and processing vast volumes of satellite imagery and geospatial datasets. The Earth Engine Data Catalog is a valuable resource for researchers and practitioners in remote sensing, environmental monitoring, climate studies, land management, and other geospatial applications [ 5 , 49 ] . Combining spatial data, such as elevation maps, with structured or unstructured data, such as text, financial metrics, or time series, can substantially enhance predictive modeling across various domains, including finance. For example, Stevenson et al. [ 16 ] examined the effectiveness of LiDAR-generated elevation maps in identifying socio-economic indicators. Using deep learning techniques, their study demonstrated that LiDAR maps can be transformed into meaningful embeddings and that these images alone can model seven indicators of sociodemographic deprivation. Furthermore, Borochov et al. [ 6 ] proposed that deprivation indices encompass a wide range of socioeconomic markers, which can potentially reflect disparities in the ability to meet mortgage obligations across different regions. This insight motivates the exploration of publicly available geospatial data, such as LiDAR-based maps, to identify regional mortgage risk patterns and enhance predictive capabilities in mortgage default modeling. Another source of unstructured and publicly available media that plays a critical role in finance is news [ 7 ] . Financial news provides timely and valuable information that can influence investment decisions, market trends, and overall financial strategies. News related to economic indicators, company earnings, policy changes, and other macroeconomic factors can influence investor sentiment and significantly shape market movements [ 7 ] . Traders, investors, and financial professionals routinely monitor news outlets to stay informed about emerging developments and adjust their strategies accordingly. The ability to interpret and analyze news within a financial context can guide investment choices, inform risk management, and improve decision-making processes. The mortgage market, in particular, is highly sensitive to economic fluctuations, market sentiment, and policy changes [ 51 ] . For example, unexpected news of a potential economic downturn can increase the perceived risk of default by lenders, leading them to tighten lending standards or reduce the availability of mortgage credit [ 52 ] . In contrast, positive news indicating strong economic conditions or favorable policy changes can lower perceived risk, prompting lenders to ease lending criteria and offer more competitive

Read the complete text