RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation

Haofeng Wang, Yu Zang · 2025

Read RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation by Haofeng Wang, Yu Zang (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06899.txt.

Introduction

Haofeng Wang and Yu Zhang argue that judging vision-language models only by their final answers misses whether the reasoning was sound. A model can reach the right answer through flawed steps. Their metric, the Reasoning Process Tree Score, organizes a model's reasoning steps into a tree and scores the faithfulness of each step using the tree's structure.

Weights adjust dynamically, so the metric grades the whole chain and pinpoints where it breaks. To test it, the authors built RPTS-Eval, a benchmark of 374 images and 390 reasoning instances with reliable visual and textual clues as leaf nodes. Evaluating models such as GPT-4o and LLaVA-Next exposed weaknesses in multimodal reasoning and gaps between open-source and commercial systems.

Catalog record

CREATORHaofeng Wang, Yu Zang
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE29 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2511-06899

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

RPTS: Tree-Structured Reasoning Process Scoring for Faithful Multimodal Evaluation Haofeng Wang, Yu Zang Abstract Large Vision-Language Models (LVLMs) excel in multimodal reasoning and have shown impressive performance on various multimodal benchmarks. However, most of these benchmarks evaluate models primarily through multiple-choice or short-answer formats, which do not take the reasoning process into account. Although some benchmarks assess the reasoning process, their methods are often overly simplistic and only examine reasoning when answers are incorrect. This approach overlooks scenarios where flawed reasoning leads to correct answers. In addition, these benchmarks do not consider the impact of intermodal relationships on reasoning. To address this issue, we propose the Reasoning Process Tree Score (RPTS), a tree structure-based metric to assess reasoning processes. Specifically, we organize the reasoning steps into a reasoning tree and leverage its hierarchical information to assign weighted faithfulness scores to each reasoning step. By dynamically adjusting these weights, RPTS not only evaluates the overall correctness of the reasoning, but also pinpoints where the model fails in the reasoning. To validate RPTS in real-world multimodal scenarios, we construct a new benchmark, RPTS-Eval, comprising 374 images and 390 reasoning instances. Each instance includes reliable visual-textual clues that serve as leaf nodes of the reasoning tree. Furthermore, we define three types of intermodal relationships to investigate how intermodal interactions influence the reasoning process. We evaluated representative LVLMs (e.g., GPT4o, Llava-Next), uncovering their limitations in multimodal reasoning and highlighting the differences between open-source and closed-source commercial LVLMs. We believe that this benchmark will contribute to the advancement of research in the field of multimodal reasoning. Code — https://aaai.org/example/code Datasets — https://aaai.org/example/datasets Extended version — https://aaai.org/example/extended-version Introduction Recent advances in multimodal foundation models have demonstrated increasingly sophisticated capabilities in the combination of visual and textual information ( 17 ) . However, as these models begin to assist in evidentiary reasoning tasks such as criminal case analysis - where establishing reliable connections between surveillance footage (visual modality), forensic reports (textual modality), and other evidence is crucial, and where conclusions must follow rigorous, verifiable reasoning chains - two critical questions emerge: 1. Can current evaluations distinguish between logically valid reasoning and coincidentally correct conclusions? 2. Do existing frameworks capture the non-linear, cross-modal reasoning required to resolve conflicting evidence? Figure 1: Comparison between Unimodal, existing Multimodal benchmarks and our RPTS. Left : Current multimodal benchmarks fail to detect instances where reasoning errors are present, yet the answer remains correct. Right : The unimodal approach is unable to handle reasoning involving conflicting information across different modalities. Most existing benchmarks focus solely on task accuracy through multiple choice or short answer formats ( 20 ; 21 ) , completely ignoring the reasoning process. This approach fails to detect when models arrive at correct conclusions through flawed reasoning, a phenomenon we call ”right answers for wrong reasons” , as illustrated on the left side of Figure 1 . The few works that examine reasoning processes ( 9 ; 18 ) typically adopt an oversimplified linear evaluation framework. These approaches are fundamentally mismatched to the complex, non-linear nature of real-world reasoning, where multimodal evidence may appear conflicting yet collectively support valid conclusions. To address these limitations, we introduce a novel evaluation metric: Reasoning Process Tree Score (RPTS), designed to assess multimodal reasoning processes. The core innovation of RPTS lies in its tree-structured representation of reasoning, where leaf nodes correspond to atomic evidence units (visual or textual) and non-leaf nodes capture the hierarchical derivation of intermediate conclusions. This structure inherently accommodates the non-linear interactions characteristic of multimodal reasoning. Furthermore, RPTS incorporates two key hyperparameters whose adjustable values enable precise quantification of both global and local logical consistency, thereby facilitating accurate error localization within the reasoning chain. Figure 2: An example of RPTS-Eval. To support comprehensive evaluation using RPTS, we developed the RPTS-Eval benchmark comprising 390 carefully constructed reasoning instances. Figure 2 shows examples of RPTS-Eval. Each instance contains complete and reliable multimodal atomic evidence for building reasoning trees. To systematically investigate how inter-modal relationships affect reasoning, we defined three distinct modality interaction types: guided (related without interference), adversarial (related with interference), and independent (unrelated), with each instance manually annotated accordingly. Experimental results on RPTS-Eval demonstrate RPTS’s capability to identify flawed reasoning and precisely localize errors, while revealing significant limitations in current LVLMs’ reasoning abilities. The primary contributions of our work can be summarized as follows: • We introduce a new metric, RPTS, for detecting correct conclusions based on faulty reasoning and genuinely logical reasoning processes, reflecting both overall and local logic of reasoning, achieving error localization. • We constructed RPTS-Eval, a novel benchmark for multimodal reasoning evaluation. Compared to existing datasets, RPTS-Eval provides reliable multimodal annotations of atomic evidence that facilitate a rigorous assessment of reasoning processes. • We define three types of relationships between modalities in reasoning, which clarify the classification of multimodal reasoning. • We conducted extensive experiments with our RPTS-Eval. The results reveal that current open-source LVLMs have difficulty drawing conclusions from images for further inference and show varying performance across different languages. Related Work MLLM Evaluation Benchmarks Classic multimodal benchmarks typically assess the specific reasoning abilities of the models. For example, OK-VQA ( 16 ) evaluates a model’s capacity to leverage external knowledge for reasoning, while VCR ( 22 ) focuses on human-related common sense reasoning. To evaluate the comprehensive capabilities of a model, researchers have proposed various benchmarks, such as MMBench ( 15 ) , SEED-Bench ( 11 ) , MM-Vet ( 20 ) , and MMMU ( 21 ) . These benchmarks scrutinize the reasoning abilities of models from diverse perspectives, often employing multiple choice or simplified formats to facilitate the evaluation process. InfiMM-Eval ( 10 ) incorporates the reasoning process into the evaluation, scoring the entire reasoning process. However, it cannot perform a more detailed analysis of reasoning and its evaluation method cannot exclude cases where incorrect reasoning leads to a correct answer. Verify Reasoning Process Recent studies have introduced various techniques for evaluating reasoning processes. ROSCOE ( 9 ) proposes a set of quality metrics to assess reasoning from four perspectives: semantic alignment, semantic similarity, logical correctness, and semantic coherence. ReCEval ( 18 ) evaluates reasoning based on two criteria: whether the reasoning steps are correct and whether new information is derived from the reasoning. REVEAL provides a dataset to validate whether a model can be used to verify the reasoning process. Figure 3: The calculation process of RPTS and accuracy. RPTS The computation of RPTS consists of two stages: Reasoning Parsing and Metric Calculation. Figure 3 illustrates the workflow of RPTS alongside the accuracy computation. Reasoning Parsing To construct the reasoning tree, we first parse the model’s reasoning into a structured format: ”[PREMISE] + [PREMISE] \rightarrow [CONCLUSION]”, where ’[PREMISE]’ can be derived from visual clues, textual clues, or intermediate conclusions from prior steps. However, existing open-source MLLMs cannot strictly adhere to this output format. To address this, we first employ chain-of-thought (CoT) prompting to guide the model in generating step-by-step reasoning with explicit premises. Subsequently, we use GPT-4 to reformat the reasoning into a structured, easily parsable representation. The parsed reasoning results can also be utilized for accuracy computation. LLM-Based Scorer Now, each reasoning step in our approach strictly adheres to the ”[PREMISE] +

Read the complete text