Towards explainable language reasoning via multi-modal knowledge graphs Authors: Chunyu Lu, Jun Luo, Kang Yu, Tianran Chen, Xueli Wang, Feng Qian, Chen Xi Published: 2026 ABSTRACT AbstractThe increasing complexity of natural language reasoning in artificial intelligence necessitates a shift from opaque, black-box processing to transparent, interpretable decision-making. While large language models (LLMs) demonstrate remarkable generative capabilities, their reasoning pathways often remain implicit, which limits auditability in high-stakes settings. To address this issue, we propose a framework for explainable language reasoning grounded in Multi-Modal Knowledge Graphs (MMKGs). Our approach unifies textual, visual, and structural knowledge in a shared graph representation, enabling the system to anchor inference in explicit semantic relations rather than in latent correlations alone. We introduce a planner-executor design in which an LLM generates constrained symbolic plans and a deterministic graph engine executes them to return both an answer and a replayable explanation subgraph. Empirically, the framework is competitive on six textual and multimodal benchmarks, reaching 79.8% Hits@1 on WebQSP and 49.3 accuracy on OK-VQA under the protocol used in this study. We deliberately report the main quantitative comparisons as point estimates and interpret small margins conservatively. The principal contribution of the work is therefore the MMKG-grounded reasoning architecture, its operationally auditable explanation mechanism, and the accompanying clarification of what can and cannot be claimed theoretically about explanation faithfulness.Keywords: Explainable AI, Multi-modal knowledge graphs, Natural language processing, Interpretable machine learning, Knowledge representationSubject terms: Mathematics and computing, Psychology, Psychology Introduction Artificial intelligence (AI) has transformed how humans interact with technology, particularly through advances in natural language processing (NLP) and reasoning. Modern systems can answer complex questions, generate fluent dialogue, and integrate knowledge across multiple sources. Large language models (LLMs) have powered this progress by learning from vast corpora and encoding subtle semantic regularities. Yet their most celebrated strength-the ability to generate human-like responses-coexists with their most pressing weakness: a lack of interpretability. In high-stakes settings such as healthcare, finance, law, and scientific discovery, decision-makers cannot rely on answers that are correct most of the time but opaque all of the time. They need transparent systems that can explain how an answer was derived, cite evidence, and allow human auditors to inspect the reasoning process. As a result, explainable artificial intelligence (XAI) has become an urgent research priority1,2. Although various explanation techniques exist, attention heatmaps, saliency visualizations, post-hoc rationales, many have limited faithfulness. They often approximate what looks like reasoning rather than exposing what caused a model’s decision. For example, chain-of-thought prompting allows LLMs to produce intermediate steps, but recent studies show these rationales can be unfaithful: models may generate plausible reasoning chains that are disconnected from the underlying computation3,4. This gap between generated explanations and actual decision pathways undermines trust, as end-users cannot determine whether a rationale is faithful or fabricated. In domains where accountability is critical, such as clinical decision support or judicial assistance, the risk of misleading explanations is unacceptable. Knowledge graphs (KGs) provide a way forward. A KG encodes entities and their relationships in explicit triples, creating a semantic network that is both machine-readable and human-interpretable. When combined with machine learning, KGs can serve as external memory, constrain reasoning, and produce explicit inference paths. Methods such as KG-BERT for knowledge graph completion5, multi-hop graph reasoning networks6, and explainable QA pipelines7 illustrate the potential of KGs to add structure and interpretability. Moreover, graph neural networks (GNNs) and their explanation modules such as GNNExplainer8 provide ways to highlight subgraphs most responsible for predictions. By anchoring reasoning in a graph, explanations can emerge as traversable paths rather than abstract probabilities. Human reasoning, however, is not purely textual. People draw inferences from visual perception, contextual cues, diagrams, and structured knowledge. The last five years have seen an explosion of multimodal AI research, particularly in vision-language pretraining. Contrastive learning approaches such as CLIP9 and ALIGN10 align images and text in shared embedding spaces, enabling zero-shot classification and retrieval. Fusion-based approaches like ALBEF11, BLIP12, and BLIP-213 integrate representations to support both understanding and generation. Large multimodal language models demonstrate few-shot learning across interleaved text and image inputs. These advances underscore the importance of multimodal learning, but they remain largely correlation-based. They lack explicit relational structure and thus cannot easily expose interpretable reasoning chains. To achieve both performance and transparency, multimodal knowledge graphs (MMKGs) have emerged as a promising paradigm. MMKGs extend traditional KGs by incorporating multimodal data, images, audio, video, alongside text and structured triples. Nodes may carry multimodal representations, and edges may encode relations enriched with perceptual features. By grounding reasoning in MMKGs, systems can align evidence across modalities and trace explicit inference paths that include visual and textual elements. For example, a QA system might detect an object in an image, map it to a KG entity, and traverse relations to textual evidence, producing an answer accompanied by a multimodal explanation. Surveys by Chen et al.14,15 provide comprehensive overviews of MMKG construction, completion, and applications, highlighting their growing role in AI. The literature already provides essential building blocks. Retrieval-augmented generation (RAG)16 combines parametric models with external memories to produce factual, citable outputs. Commonsense induction frameworks like COMET17 enrich KGs with plausible relations. Vision-language models demonstrate large-scale cross-modal associations. Explainable QA over graphs7 shows how reasoning paths can be returned with answers. Yet, no existing framework integrates all three objectives simultaneously: multimodal coverage, reasoning accuracy, and faithful explanation. Current multimodal models excel at correlation but not transparency. KG-based systems provide transparency but are limited in multimodal coverage. And XAI methods provide explanations but not always faithfully. This fragmentation leaves a gap for approaches that unify these elements. The central problem this paper addresses is the lack of a unified framework that grounds language reasoning in multimodal knowledge graphs, enabling explanations that are both faithful and interpretable. Existing methods either generate plausible rationales without guarantees of faithfulness, or they rely on unimodal KGs that omit crucial perceptual evidence. Without multimodal grounding, explanations risk incompleteness; without structured graphs, explanations risk unfaithfulness; without careful reasoning design, explanations risk being inefficient or opaque. Bridging these challenges requires a framework that integrates MMKGs with explicit reasoning mechanisms capable of producing verifiable inference paths. We argue that MMKGs offer three distinctive advantages. First, they provide faithfulness at the execution level: explanations are tied to explicit graph traversals rather than post-hoc approximations. Second, they provide auditability: nodes and edges can store provenance, allowing explanations to cite their evidence sources. Third, they provide adaptability: graphs can be incrementally updated without retraining, ensuring that reasoning remains current as knowledge evolves. By coupling LLM-based reasoning with MMKG-grounded execution, we aim to balance performance with interpretability, offering explanations that are useful, verifiable, and trustworthy. Contributions. The main contributions of this work are as follows: We propose a MMKG-grounded planner-executor architecture that integrates textual, visual, and symbolic knowledge and produces executable explanation subgraphs rather than free-form post-hoc rationales. We propose a MMKG-grounded planner-executor architecture that integrates textual, visual, and symbolic knowledge and produces executable explanation subgraphs rather than free-form post-hoc rationales. We formalize a restricted operational sufficiency property for executed explanation subgraphs under deterministic replay, and we explicitly distinguish this execution-level guarantee from broader human-centered notions of explanation faithfulness. We formalize a restricted operational sufficiency property for executed explanation subgraphs under deterministic replay, and we explicitly distinguish this execution-level guarantee from broader human-centered notions of explanation faithfulness. We provide an empirical study on six benchmarks together with a reproducibility package, including dataset-split documentation, baseline-source annotations, explicit operator thresholds, and an expanded notation appendix. We provide an empirical study on six benchmarks together with a reproducibility package, including dataset-split documentation, baseline-source annotations, explicit operator thresholds, and an expanded notation appendix. Related work Explainable AI for language reasoning. The need for interpretable AI has driven a surge of research in natural language explanations, post-hoc rationales, and visualization methods. Surveys such as Danilevsky et al.1 and Rajabi et al.2 provide comprehensive overviews of explanation approaches in NLP. Saliency-based and attention-based techniques attempt to reveal influential tokens but often lack causal faithfulness. Recent interest in chain-of-thought (CoT) prompting has further popularized natural language rationales, yet studies show that these explanations can be unfaithful to the actual decision process3. Lyu et al.4 propose constraints to enforce faithful CoT reasoning, but these methods still operate largely in a text-only setting. Overall, while XAI for language has advanced interpretability, it remains challenged by the faithfulness gap and the absence of structured, multimodal grounding. Knowledge graph–based reasoning. Knowledge graphs (KGs) encode entities and relations in explicit triples, providing a natural substrate for interpretable reasoning. Early efforts such as KG-BERT5 adapted pretrained language models for knowledge graph completion, while Feng et al.6 developed multi-hop relational reasoning networks to enable scalable inference for QA. Explaining reasoning paths has been studied through symbolic query generation and graph traversal methods7. GNNExplainer8 and related work introduced explanation methods for graph neural networks, highlighting subgraphs responsible for predictions. Commonsense induction frameworks such as COMET17 demonstrate how LMs can expand KGs with new relations, enriching graph coverage. Despite these advances, most KG-based approaches focus on structured or textual data only, with limited integration of multimodal evidence. Recent work has also emphasized the broader role of structured external knowledge in language and multimodal reasoning. In particular, explainable QA pipelines over graphs show how explicit reasoning paths can be returned alongside answers7, while recent surveys on multi-modal knowledge graphs and KG-supported multimodal learning highlight the importance of graph structure for grounding, cross-modal alignment, and interpretability14,18. We position our work in this research line, with a particular focus on separating symbolic planning from deterministic execution so that the explanation corresponds to an auditable execution record. Multi-modal knowledge graphs. Recent surveys highlight the construction and application of multi-modal knowledge graphs (MMKGs), which extend KGs with textual, visual, and sometimes audio modalities14,15. MMKGs provide richer context for tasks such as entity alignment, cross-modal retrieval, and knowledge completion. Yu et al.19 demonstrated cross-modal reasoning in knowledge-based visual QA, illustrating how integrating vision and graph relations can improve interpretability. At the same time, challenges remain in scalability, cross-modal alignment, and faithful explanation generation. While MMKGs promise explicit and interpretable multimodal reasoning, systematic integration with language reasoning systems is still in its infancy. Notation We summarize all symbols and their domains in Appendix A. Throughout, entities are denoted by and their embeddings by . We reserve h, t exclusively for head/tail entity indices in triples, and use or for embeddings. All displayed equations are numbered for unambiguous referencing. Method We propose a unified framework for explainable language reasoning grounded in Multi-Modal Knowledge Graphs (MMKGs), as illustrated in Fig. 1. The framework converts heterogeneous inputs (text and images in the present experiments, with video treated as an architectural extension rather than an evaluated modality) into a single graph substrate, learns aligned multimodal representations, executes query-specific reasoning over the graph, and returns both an answer and a faithful explanation. The central design choice is to separate planning from execution: an LLM generates a constrained symbolic plan, while the final computation is carried out by deterministic graph operations. This separation ensures that explanations correspond to the executed evidence rather than post-hoc rationales. Overall architecture of the proposed framework for explainable language reasoning via multi-modal knowledge graphs (MMKGs). Framework overview Given a query Q (optionally accompanied by multimodal context such as an image or structured table), the system performs (i) grounding to MMKG entities and relations, (ii) subgraph retrieval and path search guided by symbolic constraints and learned scoring, (iii) deterministic execution to produce an answer, and (iv) extraction and verbalization of the executed reasoning subgraph as an explanation. The framework is designed around three principles: (1) structural grounding-all answers must be derived from explicit graph evidence; (2) multimodal integration-entities are represented using aligned features across modalities; and (3) faithful explanation-the explanation is the subgraph traversed during execution, optionally pruned for local minimality. Formally, the system maintains an MMKG as the shared substrate. For each query, the planner outputs a symbolic plan and the executor returns an answer A and an executed subgraph . The final user-facing explanation is constructed from , annotated with confidence, provenance, and verification signals when available. Multi-modal knowledge graph construction The first component constructs an MMKG that integrates symbolic relations with multimodal features. We define the MMKG as where is the set of entity nodes, is the set of directed relational edges, is the relation vocabulary, and maps each node to modality-specific features. denotes supported modalities (e.g., text, image, numeric), and stores per-modality features such as textual embeddings, visual descriptors, or structured attributes. This formalization allows a single entity node to carry multiple modality signatures while sharing a unified symbolic identifier. Entity and relation extraction. Graph construction begins with modality-specific extraction pipelines. For text corpora, named entity recognition and relation extraction identify candidate triples. For images and videos, object detection and scene graph generation produce visual entities and relationships; for videos, we aggregate detections across frames and optionally create temporal edges. Each extracted entity mention is resolved to a canonical node through entity linking and clustering, which combines string similarity, contextual similarity, and embedding-based matching. Cross-modal alignment. A major challenge is aligning entities across modalities. We align textual mentions and visual objects by mapping both modalities into a shared embedding space (e.g., CLIP) and maximizing cosine similarity, while enforcing type compatibility constraints from ontologies (e.g., “instrument” vs. “person”). For example, the textual entity “stethoscope” is linked to visual regions whose embeddings are closest in the shared space, then validated by type constraints. The output is a consistent entity inventory in which each node aggregates aligned features from multiple modalities. Graph enrichment and completion. After constructing the backbone, we enrich the MMKG to improve coverage. We apply KG completion to predict missing relations. For a candidate triple (h, r, t), plausibility is estimated by a translational embedding score: where are entity embeddings and is the relation embedding. High-scoring triples can be added with a confidence tag. In addition, commonsense induction models (e.g., COMET) can propose plausible but unobserved edges, which are stored with lower provenance and used cautiously during reasoning. Visual descriptors from vision-language encoders are attached to nodes to support queries requiring perceptual grounding. Dynamic updates and indexing. Real-world knowledge evolves, so the MMKG supports incremental updates. New documents, images, or videos are processed to add or update entities and relations, followed by incremental alignment to merge with existing nodes when appropriate. We maintain graph versioning to enable temporal audits and to support time-aware reasoning. For scalability, the MMKG is sharded and indexed (entity index, relation index, ANN embedding index) to enable efficient subgraph retrieval at inference time. Knowledge substrate verification Graph structure improves interpretability but does not guarantee correctness. We therefore introduce an automated verification layer that validates candidate evidence subgraphs before finalizing an answer. Each edge is annotated with provenance (source, extraction method, confidence) and timestamps. During inference, the executor produces an evidence subgraph ; the verifier checks whether the key claims in are consistent with trusted references (e.g., curated KB snapshots, domain databases, or vetted web sources). Verification operator. For each factual edge , we compute a verification score: where encodes provenance quality (e.g., curated source vs. weak extraction) and measures agreement with trusted sources. Edges failing are flagged. The system then either (i) searches alternative paths that avoid low-verification edges, or (ii) outputs an uncertainty-aware answer with explicit warnings and provenance. This layer is especially important for high-stakes settings where outdated or incorrect knowledge must be detected. Representation learning in multi-modal knowledge graphs To support reasoning and alignment, the MMKG is embedded into a shared latent space. Each node carries modality-specific features (e.g., textual embeddings , visual embeddings , numeric attributes ). We map these features into a unified representation via a fusion function: where is parameterized by . We support early fusion (concatenation), late fusion (weighted sums), and hybrid fusion. In hybrid fusion, a gating network assigns dynamic weights to each modality: Here and are learnable parameters, initialized with Xavier uniform; we apply layer-norm to each modality projection to stabilize training. We set for all experiments. Relational message passing. Graph structure is incorporated via a relational GNN. At layer , the update for node v is with relation-specific normalization After L layers, the final embedding is , which captures both multimodal features and relational context. Training objectives. We align modalities using a contrastive objective over paired text-image representations: where is cosine similarity and is a temperature. The full objective combines graph-structural learning and optional supervised task learning: We use margin , with 64 negatives per positive sampled from a smoothed degree distribution (). Reasoning mechanisms Reasoning is performed over the MMKG using a hybrid of symbolic search and learned guidance. We first retrieve a query-conditioned subgraph using grounded entities and constraints. Within , we explore candidate paths via symbolic traversal, optionally guided by a neural policy. Symbolic path search. For a grounded start entity and relation sequence , a path corresponds to an explicit reasoning chain. Rather than multiplying raw scores in probability space, we score paths in log-space using Eq. 16, which combines triple plausibility and edge confidence. The highest-scoring paths are retained for execution. Neural policy guidance. We model multi-hop reasoning as a sequential decision process. At step t, the state contains the current node and query embedding ; actions correspond to the outgoing edges in the current frontier, denoted . The policy is where and are policy parameters for the current action set. The policy is trained to maximize expected reward where if the path reaches a correct answer and 0 otherwise. In our framework, the policy guides search and ranking, while the final answer is still produced by deterministic execution. LLM planning and deterministic execution. Large language models are incorporated as high-level planners. Given query Q, the LLM outputs a symbolic plan under a restricted grammar (Appendix B), such as . The executor runs the plan over : Exec is a deterministic interpreter that executes the plan grammar in Appendix B using fixed hyperparameters (beam width , max hops depending on dataset, A* heuristic , and deterministic tie-breaking by (score, node-id, edge-id)). The executed subgraph is defined as the union of edges returned by the primitive operators along the unique execution trace T. Compositional queries and uncertainty. For compositional reasoning, we represent relation composition in embedding space: Here denotes relation composition. In the present experiments we instantiate as element-wise addition, which is consistent with a TransE-style composition rule. Circular convolution is retained only as an architectural alternative and is not claimed as an evaluated ablation. Candidates are ranked by For a path P with edges , the final score is computed stably in log-space: This formulation improves robustness to long-horizon reasoning and enables calibrated explanation ranking. MMKG-grounded hybrid reasoning Implementation details and entity alignment We implement multimodal alignment by combining CLIP-based text-image similarity, string similarity, and KG-type compatibility constraints. Fusion uses the gating formulation above, and incremental updates integrate newly discovered nodes and edges while preserving node identity through alignment. Entity alignment procedure Operational sufficiency and its limits We formalize a restricted execution-level property of the proposed planner-executor framework. Let denote a deterministic transition system that executes a symbolic plan over graph , where A is the returned answer, is the ordered operator trace, and is the set of edges read by the operators in T. Assumption 1 (Deterministic replay). Given fixed hyperparameters, deterministic tie-breaking, and a fixed adjacency/index state, each primitive operator returns the same output whenever called with the same input state. Proposition 1 (Operational sufficiency under replay). Let denote the set of edges read during the trace T produced by . Under Assumption 1, Interpretation. Proposition 1 establishes only that the executed subgraph is sufficient for deterministic replay of the computation. It does not prove that is semantically complete, cognitively minimal, or faithful to human intent. These broader properties remain empirical questions and are evaluated in this work only through operational metrics such as answer-preservation sufficiency and edge deletion sensitivity. Practical implication. The proposition justifies storing as an auditable execution record. It should therefore be interpreted as a reproducibility guarantee for the computation, not as a complete theory of explanation quality. Feedback-based refinement Human evaluation provides preference signals beyond end-task accuracy. Following instruction-tuning with human feedback20 and recent work on aggregating crowd-sourced feedback for RLHF in code generation21, we formalize pairwise preferences over explanations and optimize a reranker for path selection: where scores coherence, sufficiency, and provenance transparency. This module updates (i) the neural policy for edge selection, and (ii) the explanation verbalizer, resulting in a self-improving system. Ethics, safety, and societal impact The proposed framework improves transparency by exposing explicit evidence paths, but explainability does not eliminate ethical and societal risks. We therefore incorporate provenance, uncertainty, and verification signals, and we outline deployment safeguards. Bias and fairness. MMKGs inherit biases from their sources (coverage imbalance in encyclopedic KBs, demographic skew in vision datasets). Explanations may inadvertently legitimize biased evidence by making it appear authoritative. We mitigate this by attaching provenance and confidence metadata to edges, encouraging source diversity during retrieval, and recommending stratified evaluation of both answers and explanations. Explanations should be interpreted as auditable evidence traces rather than guarantees of fairness. Privacy and data protection. Multimodal inputs may contain sensitive personal information. Incorporating such content into an MMKG increases re-identification risk due to relational linking. Practical safeguards include data minimization (storing embeddings instead of raw media when feasible), access control for sensitive nodes/edges, automated anonymization and redaction, retention and deletion policies, and audit logs for compliance with applicable regulations. Trust calibration and over-reliance. Faithful explanations may increase user trust beyond warranted levels. To calibrate trust, we expose edge confidence , verification score , and unverifiable flags when evidence cannot be validated. We recommend user interfaces that distinguish verified from speculative edges and require human review in high-stakes settings. Robustness and misuse. Adversaries may attempt to manipulate the MMKG (poisoning relations or alignments) to induce plausible but misleading explanations. Mitigations include restricting high-impact relations to curated sources, versioned updates with anomaly detection, cross-checking critical claims against trusted references, and limiting the influence of low-provenance edges during scoring. Governance mechanisms should prevent fully autonomous decision-making in sensitive applications. Environmental impact and limitations. Training multimodal encoders, GNNs, and using LLM planning increases compute and energy usage. We recommend mixed-precision training, subgraph-limited inference, and caching to reduce overhead. Finally, the framework does not guarantee correctness: its reliability is bounded by MMKG quality, alignment accuracy, and verification coverage. We therefore position it as an assistive, auditable reasoning tool rather than an autonomous decision-maker. Evaluation Datasets and benchmarks We evaluate the framework on a diverse set of benchmarks covering textual, visual, and multimodal reasoning (Table 1). Table 1Dataset split and evaluation protocol summary used in this study.DatasetSplit usedProtocol noteSourceWebQSPOfficial train/test splitEvaluated on the standard KGQA split22MetaQAOfficial 1-hop/2-hop/3-hop train/dev/test splitWe report the 3-hop benchmark in Table 223ComplexWebQuestionsTrain/dev/testStandard CWQ split following prior work24OK-VQAOfficial train/validation/test splitOpen-ended visual QA with external knowledge25ScienceQAOfficial 60/20/20 splitMultimodal multiple-choice reasoning26ImageNet-A + captionsStandard ImageNet-A evaluation set with paired captionsRobustness evaluation only27 Dataset split and evaluation protocol summary used in this study. - WebQSP22: a widely used benchmark for question answering over Freebase, requiring multi-hop reasoning. - MetaQA23: a dataset with synthetic questions over movie knowledge graphs, including 1-hop, 2-hop, and 3-hop queries. - ComplexWebQuestions24: a dataset that contains compositional questions over the web and requires logical composition of relations. - OK-VQA25: an open-ended visual question answering dataset requiring external knowledge. - ScienceQA26: a multimodal benchmark for scientific reasoning using both text and images. - ImageNet-A with captions27: adversarial images annotated with textual descriptions, testing robustness. Experimental setup Baselines We compare against four baseline families: LLM-only reasoning: direct QA using GPT-style LLMs28,29. LLM-only reasoning: direct QA using GPT-style LLMs28,29. KG-only reasoning: symbolic or embedding-based reasoning over knowledge graphs, including TransE-style approaches30 and graph reasoning models31. KG-only reasoning: symbolic or embedding-based reasoning over knowledge graphs, including TransE-style approaches30 and graph reasoning models31. Multimodal fusion without KG: vision-language models such as BLIP-213. Multimodal fusion without KG: vision-language models such as BLIP-213. Explainable QA / retrieval baselines: Neural Symbolic Machines32 and retrieval-augmented generation16. Explainable QA / retrieval baselines: Neural Symbolic Machines32 and retrieval-augmented generation16. Implementation details Our MMKG construction integrates entities from Wikidata (as symbolic backbone), Wikipedia (as textual evidence), and MS-COCO (as visual grounding). Entity alignment is achieved using CLIP embeddings. Graph embeddings are learned using a 3-layer multimodal graph neural network with hidden dimension . Training is performed with Adam optimizer, learning rate , batch size 128, and dropout 0.3. Contrastive alignment is trained with temperature . Reasoning uses a hybrid symbolic-neural approach: symbolic traversal scores candidate paths, while an LLM generates high-level reasoning plans. Explanations are generated from reasoning subgraphs and verbalized via template-based natural language generation. Experiments are conducted on an NVIDIA A100 GPU with 80GB memory. Quantitative evaluation Answer accuracy Accuracy is evaluated using Hits@K and F1 score. For each query Q, let be the set of predicted answers and the ground truth. Precision, recall, and F1 are defined as Hits@K measures whether the correct answer is among the top K ranked predictions. Results show that our framework achieves state-of-the-art F1 and Hits@10 across both text-only and multimodal QA datasets. Faithfulness metrics We distinguish between metrics that are broadly applicable and metrics that require executable silver paths. Answer-Preservation Sufficiency (APS): Answer-Preservation Sufficiency (APS): Edge Deletion Sensitivity (EDS): the fraction of edges whose removal changes the returned answer. Edge Deletion Sensitivity (EDS): the fraction of edges whose removal changes the returned answer. Path Coverage: reported only when a silver executable path is available (e.g., WebQSP/MetaQA-style settings); it is not reported for OK-VQA or ScienceQA. Path Coverage: reported only when a silver executable path is available (e.g., WebQSP/MetaQA-style settings); it is not reported for OK-VQA or ScienceQA. Accordingly, Path Coverage should be interpreted only on datasets with executable silver paths, whereas APS and EDS are the primary operational metrics for broader evaluation. Multimodal alignment scores Alignment quality between text and images is evaluated using recall@K in retrieval tasks. Given a textual query, the system retrieves candidate images; recall@K is defined as Our approach consistently outperforms fusion-only baselines, demonstrating the importance of graph structure in multimodal reasoning. Statistical significance We perform paired bootstrap resampling (10,000 iterations) to compare accuracy and faithfulness metrics against baselines. Results are statistically significant with across all datasets. We report mean ± standard deviation over 5 random seeds for all learned components, and 95% bootstrap confidence intervals for test metrics. For each dataset, we specify the exact baseline pairwise comparisons and apply Holm-Bonferroni correction for multiple testing. Overall performance comparison We first present a comprehensive comparison of our proposed framework with representative baselines across all six benchmark datasets. We evaluate using standard metrics: Hits@1 and Hits@10 for knowledge graph benchmarks, exact match (EM) and F1 for text-based question answering, and accuracy for multimodal reasoning tasks. Table 2 summarizes the results. The results clearly demonstrate the advantages of our framework. On textual benchmarks such as WebQSP and ComplexWebQuestions, the integration of multimodal knowledge graphs with language reasoning yields consistent gains over LLM-only and KG-only methods. For instance, our system improves Hits@1 on WebQSP by more than 5 percentage points compared to Neural Symbolic Machines and Retrieval-Augmented Transformers, highlighting the benefits of explicit graph grounding. On multimodal datasets such as OK-VQA and ScienceQA, our framework substantially outperforms vision-language baselines such as BLIP-2, confirming that multimodal reasoning benefits from structured knowledge integration rather than pure feature fusion. Accuracy gains are particularly strong on OK-VQA, where questions explicitly require external knowledge that our MMKG provides. The robustness of our approach is further confirmed on ImageNet-A with captions, a dataset designed to stress-test reasoning under adversarial visual inputs. Our framework improves accuracy by over 7 points compared to BLIP-2, suggesting that grounding reasoning in explicit structure mitigates the brittleness of end-to-end multimodal models. Overall, these comparisons validate that the combination of symbolic graph structure, multimodal alignment, and LLM-based planning enables our framework to achieve state-of-the-art performance while maintaining interpretability. Table 2Overall performance comparison on textual, visual, and multimodal reasoning benchmarks.ModelWebQSPMetaQA-3hopComplexWebQOK-VQAScienceQAImageNet-ASource typeLLM-only reasoning28,2967.461.256.838.569.242.1CitedKG-only reasoning30,3171.978.459.6–––CitedMultimodal fusion w/o KG13–––44.773.146.9CitedNeural symbolic machines3272.675.961.5–––CitedRetrieval-augmented generation1674.377.163.741.870.644.2CitedOurs (MMKG framework)79.884.269.149.377.452.6ReproducedValues are reported as point estimates under the protocol described in “Experimental setup”. The last column indicates whether a value was reproduced under our implementation or taken from the cited source. We therefore avoid formal significance claims based on these entries alone. Overall performance comparison on textual, visual, and multimodal reasoning benchmarks. Values are reported as point estimates under the protocol described in “Experimental setup”. The last column indicates whether a value was reproduced under our implementation or taken from the cited source. We therefore avoid formal significance claims based on these entries alone. Faithfulness evaluation We evaluate explanation faithfulness using four complementary metrics: (1) Path Coverage (when silver paths are available), (2) Causal Sufficiency, (3) Faithful F1, and (4) Edge Deletion Sensitivity (EDS). EDS measures the fraction of edges whose removal changes the predicted answer, quantifying the structural dependence of the decision on the explanation subgraph. As shown in Fig. 2, our MMKG-grounded execution achieves consistently higher structural dependence and sufficiency, indicating that explanations are not merely post-hoc rationales but operationally necessary for answer generation. Table 3 summarizes representative operational faithfulness scores. Because executable silver paths are unavailable for some datasets, Path Coverage is reported only where such supervision exists, whereas APS and EDS-style operational dependence are the main quantities of interest. Faithfulness comparison including path coverage, causal sufficiency, faithful F1 (scaled), and edge deletion sensitivity (EDS). Table 3Representative operational faithfulness comparison.ModelPath coverageCausal sufficiency/APSFaithful F1RAG-style retrieval baseline0.660.5560.1Ours (MMKG)0.780.7167.3Reported only where executable silver paths are available.Path Coverage is reported only on datasets with executable silver paths; APS and EDS-style dependence metrics are applicable more broadly. Representative operational faithfulness comparison. Reported only where executable silver paths are available. Path Coverage is reported only on datasets with executable silver paths; APS and EDS-style dependence metrics are applicable more broadly. MMKG construction quality evaluation We assess the quality of the constructed MMKG using entity alignment accuracy, relation extraction F1, cross-modal retrieval recall, and graph completeness. These measures reflect how well multimodal information is integrated before reasoning. The results indicate that the graph provides sufficiently accurate multimodal grounding for subsequent reasoning and explanation generation (Table 4; Figs. 3, 4, 5). Table 4MMKG construction quality evaluation.MetricScoreEntity alignment accuracy86.3%Relation extraction F182.7%Text-image recall@1074.8%Graph completeness ratio91.2% MMKG construction quality evaluation. WebQSP Hits@1 with mean ± standard deviation over 5 random seeds. Error bars show 95% bootstrap confidence intervals. OK-VQA accuracy with mean ± standard deviation over 5 seeds. ScienceQA accuracy with mean ± standard deviation over 5 seeds. Ablation study To further understand the contribution of each component in our proposed framework, we conduct an ablation study by systematically disabling individual modules. The tested variants include: (i) removing the Graph Grounding module, (ii) removing the Multimodal Alignment mechanism, (iii) removing the LLM-based Planning component, and (iv) removing the Explanation Module. We evaluate these variants on three representative benchmarks: WebQSP (Hits@1), ComplexWebQuestions (F1), and ScienceQA (Accuracy). The results are illustrated in Fig. 6. Ablation study on major components of the proposed framework. Performance is reported on WebQSP (Hits@1), ComplexWebQuestions (F1), and ScienceQA (Accuracy). Each bar corresponds to the full model or a variant with one module removed. From Fig. 6, several important insights emerge. First, the removal of Graph Grounding produces the sharpest performance degradation across all datasets. Accuracy on WebQSP falls from 79.8 to 72.3, and ScienceQA accuracy drops from 77.4 to 70.1. This decline demonstrates that explicit symbolic grounding is not simply a supporting feature but a structural necessity. Without graph grounding, the system is forced to rely on loosely coupled embeddings, which fail to capture the compositional semantics needed for robust multi-hop reasoning. Second, eliminating the Multimodal Alignment module primarily affects multimodal benchmarks. On ScienceQA, accuracy decreases by more than 6 points, confirming that cross-modal consistency is vital for correctly linking textual entities with their corresponding visual evidence. Although performance on WebQSP and ComplexWebQuestions remains relatively stable, the degradation on ScienceQA shows that alignment plays a decisive role when reasoning depends on multimodal integration. Third, removing the LLM-based Planning module significantly reduces compositional reasoning ability. The F1 score on ComplexWebQuestions falls from 69.1 to 64.2, a drop of nearly 5 points. This suggests that while symbolic graph traversal can capture basic relations, the planning ability of large language models is indispensable for handling complex queries that involve logical conjunctions, negations, or nested compositions. Finally, the absence of the Explanation Module has minimal effect on predictive accuracy, with less than 1-point difference across datasets. However, this component is essential for generating faithful and human-readable explanations. Without it, the system cannot provide interpretable reasoning chains, thus undermining user trust and transparency even though factual correctness is preserved. Robustness study While accuracy and faithfulness are essential for evaluating reasoning systems, robustness under adversarial or noisy conditions is equally critical for real-world deployment. To assess robustness, we evaluate our framework on two challenging benchmarks: ImageNet-A with captions27, which contains adversarially selected images that frequently mislead standard vision models, and OK-VQA25, an open-ended multimodal benchmark requiring reasoning over noisy and incomplete evidence. We compare our MMKG-based framework with representative cited baselines including BLIP-213, LLM-only reasoning28,29, and retrieval-augmented generation16. The results are summarized in Fig. 7. Robustness study on adversarial and noisy datasets. We evaluate on ImageNet-A (adversarial images with captions) and OK-VQA (noisy multimodal reasoning). Our MMKG framework demonstrates superior resilience compared with BLIP-2, LLM-only reasoning, and RAG. From Fig. 7, several important observations emerge. First, on ImageNet-A, our framework achieves an accuracy of 52.6 under the protocol summarized in Table 2, outperforming the cited BLIP-2 baseline (46.9) and the LLM-only baseline (42.1). These results suggest that symbolic graph grounding may help filter adversarial noise by enforcing structured reasoning chains that are less sensitive to superficial pixel-level perturbations. Second, on OK-VQA, our framework reaches 49.3, again exceeding the cited BLIP-2 and retrieval-augmented baselines in the present comparison. This pattern is consistent with the view that multimodal alignment within the MMKG helps link visual evidence to textual entities under incomplete or ambiguous conditions. Unlike fusion-only models, our approach leverages symbolic constraints to resolve uncertainty and generate explanations grounded in explicit reasoning paths. Taken together, these results suggest that robustness is not simply a byproduct of scaling language or vision models. Instead, it may emerge from the deliberate integration of symbolic structure, multimodal alignment, and faithful reasoning paths. Because these results are reported here as point estimates rather than fully documented seed-averaged statistics, we interpret the margins conservatively and view the robustness study as supportive rather than definitive. Case studies We present qualitative examples to illustrate how the model generates explanations grounded in multimodal evidence. In ScienceQA, when given an image of a magnifying glass focusing sunlight, the system identifies the visual object, aligns it to its MMKG entity, and follows a reasoning path through nodes corresponding to concentrated light and thermal ignition. In OK-VQA, the model detects a graduated cylinder and retrieves usage-related edges to justify the answer. These examples confirm that explanation paths correspond directly to the executed graph traversal. Computational cost This section analyzes the computational cost of the proposed MMKG-grounded hybrid reasoning framework, covering offline graph construction, representation learning, and online inference with explanation generation. We distinguish between offline preprocessing costs, which are amortized over many queries, and online costs, which directly affect deployment latency. Offline MMKG construction. Let denote the number of textual documents, the number of images, and the number of unique entities extracted across modalities. Textual entity and relation extraction scales as , where is the per-document extraction cost. Visual processing, including object detection and scene graph generation, scales as . Multimodal entity alignment is performed using embedding similarity with approximate nearest-neighbor (ANN) search. With standard ANN indices (e.g., HNSW or IVF), index construction costs and average query-time alignment is sublinear, typically in practice. Optional knowledge graph completion adds , where is the number of edges and is the cost of scoring candidate triples. Representation learning. Graph representation learning is performed with a relational graph neural network (GNN). For a graph with nodes, edges, embedding dimension d, and L layers, a forward pass has complexity under sparse message passing. Multimodal fusion introduces an additional cost per pass, where is the number of modalities. Contrastive multimodal alignment with batch size B incurs similarity computations, mitigated in practice via in-batch negatives, temperature scaling, and mixed-precision training. Online reasoning and explanation. At inference time, the system retrieves a query-conditioned subgraph with K candidate entities and edges. Retrieval using indexed neighborhood expansion costs . Reasoning over is implemented via bounded beam or A* search over H hops with beam width B, yielding worst-case complexity , where is the maximum out-degree within . Explanation pruning for local minimality may require multiple executions, but each execution operates on a progressively smaller subgraph and is efficiently cached. The use of a large language model as a planner adds a cost proportional to the number of generated planning tokens; this overhead is controlled by a restricted grammar and aggressive plan caching. Overall, the framework is designed so that expensive operations occur offline, while online reasoning is confined to small, retrieved subgraphs, enabling scalability to large MMKGs through indexing, sharding, and caching. Discussion This work presents a unified framework for explainable language reasoning grounded in multi-modal knowledge graphs (MMKGs), with the goal of reconciling strong reasoning performance and faithful interpretability. The experimental results indicate that explicit graph grounding can improve both accuracy and robustness on knowledge-intensive and multimodal benchmarks, while simultaneously producing explanations that are auditable and verifiable. Interpretability and performance. A key observation is that enforcing structural constraints through MMKGs does not necessarily degrade performance. On the contrary, constraining reasoning to semantically meaningful graph paths often improves multi-hop reasoning by reducing spurious correlations. However, this benefit comes with a trade-off: when the MMKG is incomplete or noisy, strict grounding may limit recall compared to fully neural models that can rely on latent associations. This highlights an inherent tension between faithfulness and coverage, suggesting that future systems should adaptively relax constraints when evidence is insufficient, while clearly signaling uncertainty to users. Limitations. Despite its advantages, the framework has several limitations. First, it relies on accurate cross-modal entity alignment, which remains imperfect and can propagate errors through the reasoning process. Second, the quality of explanations is bounded by the completeness and correctness of the underlying MMKG; missing or biased knowledge cannot be recovered through reasoning alone. Third, the use of large language models as planners introduces additional computational cost and potential failure modes when generated plans are poorly grounded or ambiguous. Future work. Several directions for future research emerge from this study. Extending MMKGs to incorporate richer modalities such as video, audio, and temporal information would enable more comprehensive reasoning. In addition, integrating continuous human feedback into the reasoning and explanation pipeline could further align explanations with human judgment over time. Finally, exploring more efficient planning and verification mechanisms will be essential for deploying MMKG-grounded reasoning systems at scale in real-world, high-stakes applications. Conclusion In this work, we introduced a unified framework for explainable language reasoning via Multi-Modal Knowledge Graphs (MMKGs). The proposed architecture integrates structured knowledge representation, multimodal alignment, and language-based reasoning into a coherent pipeline that produces not only accurate answers but also faithful and interpretable explanations. By grounding reasoning in explicit graph structures while leveraging the generative planning capabilities of large language models, our approach bridges the gap between symbolic reasoning and neural perception. Through extensive evaluation across diverse benchmarks-including WebQSP, MetaQA, ComplexWebQuestions, OK-VQA, ScienceQA, and ImageNet-A-we demonstrated that our framework consistently outperforms strong baselines in terms of accuracy, faithfulness, and robustness. The ablation study confirmed that each module of the framework contributes complementary strengths: graph grounding ensures structural fidelity, multimodal alignment provides cross-modal consistency, LLM-based planning enhances compositional reasoning, and the explanation module delivers human-understandable justifications. Moreover, robustness experiments revealed that our method is significantly more resilient to adversarial noise and incomplete evidence compared with fusion-only or retrieval-augmented models. The ability to deliver faithful reasoning chains grounded in multimodal evidence makes the proposed MMKG framework particularly suitable for real-world applications where trust, transparency, and accountability are essential, such as healthcare, scientific discovery, and decision support systems. While the current work primarily focuses on text and images, future research can extend MMKG construction to incorporate additional modalities such as audio, video, and sensor data. Another promising direction is the exploration of continual learning and temporal reasoning over evolving knowledge graphs, enabling the system to adapt dynamically to new information streams. Supplementary Information Supplementary Information. Expanded symbol table The symbol table is shown in Table 5. Table 5Expanded notation table with domains, tensor shapes, and operational values where fixed in the present study.SymbolMeaningDomain/shapeOperational value in this paperMulti-modal knowledge graph–Nodes, edges, relation vocabularySets–Supported modalitiesFinite set{Text, image} in reported experimentsModality-m feature for node vModality-dependentModality projection matrixXavier uniform initGating vector for modality mXavier uniform initModality gate weight[0, 1]Softmax-normalizeddHidden dimensionInteger256LNumber of GNN layersInteger3Self and relation transformsLearnedNormalization constant in Eq. (7)Positive integerDefined by indegree countPolicy projection matrixLearnedPolicy biasLearnedContrastive temperaturePositive scalar0.07Verification thresholdScalar in [0, 1]0.60Loss weightsNonnegative scalarsTask-dependent; tuned on validation setMargin in Positive scalar1.0BBeam widthPositive integer20HMax reasoning hopsPositive integer2 or 3 depending on datasetEdge confidence score(0, 1]Source-dependentVerification score[0, 1]Eq. (9) in revised textRelation-composition operatorBinary operator on Element-wise addition in reported experimentsKG margin lossScalarDefined in Eq. (8)Contrastive alignment lossScalarDefined in Eq. (7)Optional supervised task lossScalarUsed when labels are available Expanded notation table with domains, tensor shapes, and operational values where fixed in the present study. Plan grammar Author contributions Chunyu Lu conceived the research idea, designed the study framework, and coordinated overall project execution. Jun Luo developed methodology, implemented core algorithms, and contributed to experimental design. Kang Yu conducted experiments, performed model training, and managed technical validation. Tianran Chen collected and preprocessed datasets, curated benchmarks, and supported evaluation. Xueli Wang led data analysis, performance benchmarking, and interpretation of results; contributed substantially to manuscript revisions. Feng Qian assisted in system integration, visualization of results, and preparation of figures/tables. Chen Xi supervised the project, provided critical insights, secured resources, and finalized the manuscript for submission. Funding This work was partially supported by the key technologies for major risk prevention, control, and safety assurance of the China Russia pipeline (National Key R&D Program of China, 2022YFC30701). Data availability The datasets used and/or analysed during the current study available from the corresponding author on reasonable request. Declarations Competing interests The authors declare no competing interests. Footnotes Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations. Supplementary Information The online version contains supplementary material available at 10.1038/s41598-026-51127-4. References Associated Data Supplementary Materials Supplementary Information. Data Availability Statement The datasets used and/or analysed during the current study available from the corresponding author on reasonable request.