Cascaded Information Disclosure for Generalized Evaluation of Problem Solving Capabilities
Yunxiang Yan, Tomohiro Sawada, Kartik Goyal · 2025
Read Cascaded Information Disclosure for Generalized Evaluation of Problem Solving Capabilities by Yunxiang Yan, Tomohiro Sawada, Kartik Goyal (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-23776.txt.
Introduction
Yunxiang Yan, Tomohiro Sawada, and Kartik Goyal propose a staged way to test the problem-solving skill of language models in a 2025 paper. Standard question tests grade only the final answer, which is a rough proxy for the thinking behind it. A model can guess right for wrong reasons, or fail despite sound steps.
Their frame reveals a question in stages, each stage adding part of the facts. At each stage the model answers, exposing how its reasoning builds as facts arrive. Across varied reasoning and knowledge test sets, the staged grades separated models more sharply than plain test scores did. The staged prompts also drew out better step-by-step traces from the models. The method keeps the scale and automation of standard tests.
Catalog record
| CREATOR | Yunxiang Yan, Tomohiro Sawada, Kartik Goyal |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 72 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-23776
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Cascaded Information Disclosure for Generalized Evaluation of Problem Solving Capabilities Yunxiang Yan, Tomohiro Sawada, Kartik Goyal Abstract While question-answering (QA) benchmark performance is an automatic and scalable method to compare LLMs, it is an indirect method of evaluating their underlying problem-solving capabilities. Therefore, we propose a holistic and generalizable framework based on cascaded question disclosure that provides a more accurate estimate of the models’ problem-solving capabilities while maintaining the scalability and automation. This approach collects model responses in a stagewise manner with each stage revealing partial information about the question designed to elicit generalized reasoning in LLMs. We find that our approach not only provides a better comparison between LLMs, but also induces better intermediate traces in models compared to the standard QA paradigm. We empirically verify this behavior on diverse reasoning and knowledge-heavy QA datasets by comparing LLMs of varying sizes and families. Our approach narrows the performance gap observed in the standard QA evaluation settings, indicating that the prevalent indirect QA paradigm of evaluation overestimates the differences in performance between models. We further validate our findings by extensive ablation studies. Figure 1: Demonstration of MCQA instantiation of our Cascaded Information Disclosure evaluation framework against Standard evaluation and LLM-based Subjective evaluation of problem-solving capabilities. The standard evaluation paradigm picks the correct option but has poor justification for the answer. Cascaded Information Disclosure strategically generalizes the question and exposes parts of it to the model in a stagewise manner. The free-form response generated during ideation phase reflects model’s poor problem-solving ability. LLM-based subjective evaluation is an appealing alternative to judge the ideated response but fails in the example shown – it supports the trivial conclusion in the incorrect ideation. Verifiable Projection while objective and automatic, correctly penalizes the incorrect ideated response by picking the wrong option. 1 Introduction While general-purpose LLMs have become ubiquitous today, evaluating them holistically remains a massive challenge. Primarily, these models are compared against each other by their performance on a handful of benchmark tasks and datasets Liang et al. 2023 ; Srivastava et al. 2023 ; Wang et al. 2019 ; Sarlin et al. 2020 dominated by various kinds of question-answering tasks Hendrycks et al. 2021 ; Lin et al. 2022 ; Rajpurkar et al. 2016 ; Rein et al. 2023 . While objective and scalable, to estimate the underlying problem-solving capabilities of models, this approach is indirect – merely judging a model on its ability to pick the correct choice among distractors or to produce a correct numerical answer to a mathematical problem doesn’t measure the quality of strategies these models use to arrive at that answer. There is a pressing need Alzahrani et al. 2024 ; Gan et al. 2024 ; Li et al. 2024b for direct evaluation of problem-solving capabilities as they are better aligned to the actual use-cases of LLMs. For example, while the GPQA dataset Rein et al. 2023 contains multiple-choice questions (MCQs) that indirectly test the graduate-level knowledge of the models, the corresponding real-world use-case is more subjective – the model should possess the ability to have an accurate open-ended conversation with the user about advanced concepts. This ability is better reflected in the strategies that the LLM uses to answer GPQA questions. Hence, the focus of this paper is to directly evaluate the internal problem-solving capabilities of the models in a holistic, yet scalable and generalizable manner. We propose an evaluation framework called cascaded information disclosure which focuses on eliciting and estimating the problem-solving capabilities reflected in the intermediate traces of the models. It involves modification of standard QA tasks as follows. We first transform the question into its more generalized form and then design a scheme to partition this form into non-overlapping parts . Then we disclose these parts to the model under evaluation in a stagewise manner. While this framework accommodates multiple stages/parts, the final stage in all instantiations is a verifiable projection stage that projects the model’s responses in the earlier stages to an automatically evaluatable/verifiable form. This progressive question disclosure leads to a more accurate assessment of the subjective problem-solving capabilities of the models while the verifiable projection stage maintains the scalability and automation of evaluation. We instantiate this framework for two distinct QA modes: i) multiple-choice QA, and ii) reasoning-based math word problems. Comparing a wide variety of models of different sizes and families (§ 3.1 ), our approach narrows the performance gap observed in the standard evaluation settings, indicating that the prevalent indirect evaluation paradigm overestimates the differences in performance between models. Overall, we find that our approach not only provides a better estimate of LLMs’ underlying reasoning capabilities, but also elicits better performance and intermediate traces from the models compared to the standard QA paradigm – an effect primarily driven by the decomposition of the questions into separate foci. We further validate the soundness of our observations by extensive ablation studies. To summarize, we i) propose a novel generalized framework to test the problem-solving capabilities of LLMs that supports automation and extensibility, ii) empirically study our framework via concrete automatic instantiations across multiple QA tasks and LLMs, and iii) create manual instantiations of this framework by annotating two datasets that will be publicly available. 2 Cascaded Information Disclosure The central guiding conjecture Wood et al. 1976 ; Collins and Stevens 1983 behind our approach is that an answerer (human or machine) will employ more detailed reasoning and knowledge-based strategies and reveal more information in their answers if the question is broken down into a series of leading questions. Therefore, our approach involves exposing partial information about the question to the answerer model in a cascade of stages and eliciting potentially detailed and meaningful responses from it at each stage. More concretely, the input question Q is first converted into an abstract form \bar{Q} . The abstract question is then decomposed into n non-overlapping subsets \bar{Q}=\bigcup_{i=1}^{n}\bar{Q}_{i} . Our approach then solicits responses R_{i} from the model by presenting these subsets in a predetermined order and conditioning each response on the corresponding subset and the response in the previous stage: R_{i}\leftarrow Q_{i},R_{i-1} . The response of the final stage is considered to be the model’s answer to the full question: A(Q)=R_{n}\mid Q_{n},R_{n-1} . Crucially, the response at any stage doesn’t explicitly depend on the subparts of the question at the earlier stages – the only information the model can gather about the previous stages is through the response generated one stage prior. This promotes detailed and informative responses from the model at each stage which, as we show in our results, is the primary driver for eliciting better problem-solving strategies from the models. An appropriate decomposition procedure underpins the success of our approach and needs to be custom-designed for every instantiation of this abstract framework. In our experiments, we instantiate this framework for two different types of questions: multiple-choice questions, and math word problems. While the instantiations are procedurally different, there are some commonalities between the two instantiations. Both of them decompose the question into two stages (n=2) that we call generalized ideation and verifiable projection stages. The generalized ideation stage presents a partial generalized view of the original question to the model such that the model generates a broad and general response