DIMSUM: Discourse in Mathematical Reasoning as a Supervision Module
Krish Sharma, Niyar R Barman, Akshay Chaturvedi, Nicholas Asher · 2025
Read DIMSUM: Discourse in Mathematical Reasoning as a Supervision Module by Krish Sharma, Niyar R Barman, Akshay Chaturvedi, Nicholas Asher (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2503-04685.txt.
Introduction
Krish Sharma and co-authors add discourse structure as a coach for math word problems in a 2025 paper. Grade school math sets like GSM8k grade model thought. Fresh reads say gains there may stem from wide training soak, not true thought. The team feed models the talk frame of each problem: how its parts link in sense. Small models gain most from the frame.
One 13-billion-part model rose up to 160 percent with frame aid. Even big models that like had the set by heart gained on fresh, strange problems. Gains there ran large. The authors pitch talk frames as cheap thought help for lean models.
Catalog record
| CREATOR | Krish Sharma, Niyar R Barman, Akshay Chaturvedi, Nicholas Asher |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 43 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2503-04685
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
DIMSUM: Discourse in Mathematical Reasoning as a Supervision Module Krish Sharma, Niyar R Barman, Akshay Chaturvedi, Nicholas Asher Abstract We look at reasoning on GSM8k, a dataset of short texts presenting primary school, math problems. We find, with Mirzadeh et al. 2024 , that current LLM progress on the data set may not be explained by better reasoning but by exposure to a broader pretraining data distribution. We then introduce a novel information source for helping models with less data or inferior training reason better: discourse structure. We show that discourse structure improves performance for models like Llama2 13b by up to 160%. Even for models that have most likely memorized the data set, adding discourse structural information to the model still improves predictions and dramatically improves large model performance on out of distribution examples. \textsuperscript{\textdagger} \textsuperscript{\textdagger} footnotetext: Equal Contribution 1 Introduction Recent advancements in large language models (LLMs) have led to impressive performance on reasoning tasks, particularly on benchmark datasets like GSM8K Cobbe et al. 2021 . These models exhibit strong problem-solving abilities, often producing solutions that resemble human-like reasoning. However, recent studies have demonstrated that altering the entities or numerical values can degrade the reasoning capabilities of LLMs Mirzadeh et al. 2024 . Other studies have also shown that model reasoning is not robust Schaeffer et al. 2024 ; Valmeekam et al. 2022 ; Asher and Bhar 2024 ; Kambhampati 2024 . This raises a critical question: do these models genuinely engage in reasoning, or do they primarily rely on memorization and pattern recognition? These findings suggest a need for a deeper investigation into how LLMs process logical structures and linguistic variations. Prior research Chen et al. 2024 has also investigated the effect of permuting the premise order in GSM8K problems, showing that such modifications affect LLMs’ reasoning capabilities. This observation suggests that, unlike formal logic, where the validity of a conclusion remains unchanged regardless of premise order, natural language reasoning is shaped by structural dependencies within discourse. A linguistic perspective explains this dependency. Unlike formal logic, natural language introduces ambiguity and underspecification, which are resolved through contextual and structural cues that tell us how clauses in a text are semantically related. The ordering of clauses within a discourse, for instance, significantly impacts interpretation, and determines, in the absence of other cues, temporal and causal relationships. Consider the simple sequences in 1 from Asher 1993 : \ex . Ṗat took off his shoes. Pat got in bed. .̱ Pat got in bed. Pat took off his shoes When we permute the order of the clauses as in 1 b, most native English speakers will draw different conclusions. They will conclude from in 1 a that Pat first took off his shoes before getting into bed, whereas in 1 b that Pat took his shoes off after he got in bed. Ordering in this case determines a semantic relation– Asher 1993 calls it Narration that entails that the event in the second clause follows the event in the first. Even in seemingly simple narratives, semantic relations such as Elaboration, Background information, and Narration shape how reasoning unfolds. Additionally, discourse structure affects anaphora resolution—reordering premises can change how pronouns and referents are interpreted, thereby influencing reasoning outcomes. For example, in 1 \ex . John picked 3 apples. Sam picked 4 apples. He then picked 2 more. most native speakers would pick Sam as the referent of he in the last sentence. If we reverse the first two sentences, most speakers would pick John as the referent of he . We show that information about discourse structure improves performance for all LLMs on the GSM8k data set, but especially for older, models like Llama2 13b with poor reasoning performance by up to 160%. Even for models that have most likely memorized the data set, adding discourse structural information to the model improves performance on GSM8K and dramatically improves performance on out of distribution examples. Example of Discourse Structure: Question: Oliver picks 44 kiwis on Friday. Then he picks 58 kiwis on Saturday. On Sunday, he picks double the number of kiwis he did on Friday, but five of them were a bit smaller than average. How many kiwis does Oliver have? . Discourse Structure: <Structure> Topics: [topic-a] Oliver’s kiwi picking on consecutive days [topic-b] Sunday’s special condition (five smaller kiwis) Relationships: t_{a}\xrightarrow{\text{Elab}}t_{b} Premises: P1: Picks 44 kiwis on Friday P2: Picks 58 on Saturday P3: On Sunday picks double the Friday amount P4: Sunday has five smaller kiwis P5: Oliver wants to know total kiwis Narrative Flow: P1 P2 P3 P5 P4 narr narr narr bckgnd </Structure> Figure 1: Overview of the prompt sequence on an example from Mirzadeh et al. 2024 that GPT-o1-mini and Llama-3-8B couldn’t solve. With this prompt all models tested solved the problem correctly (see Appendix A.4.2 for model generation outputs). The full structure generation prompt ( A.3.1 ), answer generation prompt ( A.3.2 ) and few-shot examples ( A.4 ) are available in the appendix . 2 Hard GSM8K Dataset and Variants To systematically assess whether LLMs really reason or just rely on memorized patterns, we use the more difficult portion of the GSM8K data set, Hard GSM8K , (with problem statements involving 4 or more premises). We also provide a controlled variant of Hard GSM8K to create out-of-distribution (OOD) variants using the transformations defined below (see Appendix A.5 for examples). This data will be made publicly available. Unlike the data in Mirzadeh et al. 2024 , where modifications involves changing entity names and adding or subtracting premises, our variants are more complex in terms of contextual modification and number of substitutions. We investigate whether the model can maintain performance under variations in numerical values and contextual framing. This data enables us to rigorously evaluate the robustness of LLMs’ reasoning, and it serves as a test-bed to study the influence of the discourse structure. 2.1 Contextual Modification (C-MOD) In this transformation, we alter the real-world context of a problem while preserving its structural framework and numerical relationships, ensuring the modified problem shifts out of the pretraining data distribution. Given a problem statement P , we define a contextual mapping function f_{C}:P\rightarrow P^{\prime} , where entities and actions are replaced by semantically distinct counterparts from an unrelated domain. Formally, if E=\{e_{1},e_{2},\dots,e_{n}\} represents entities in the original problem and A=\{a_{1},a_{2},\dots,a_{m}\} denotes actions, then: f_{C}(E,A)=(E^{\prime},A^{\prime})\quad\text{such that }\begin{aligned} &E^{\prime}\cap E=\emptyset,\\ &A^{\prime}\cap A=\emptyset.\end{aligned} (1) For instance food items are substituted with digital files, and weight was mapped to storage size. 2.2 Numerical Modification (N-MOD) This transformation alters numerical values while preserving the problem’s reasoning structure. Given a set of numerical values N=\{n_{1},n_{2},\dots,n_{k}\} in a problem, we apply a function f_{N}:N\rightarrow N^{\prime} such that: \displaystyle\forall n_{i}\in N, \displaystyle f_{N}(n_{i})=n^{\prime}_{i}, \displaystyle\text{where}\quad\text{Scale}(N^{\prime})=\text{Scale}(N). Here, \text{Scale}(N) ensures proportionality is maintained. For instance, if the weight of an object triples, the same transformation is applied to the modified numbers. 2.3 Lexical Modification (L-MOD) This technique modifies words and phrases without altering sentence structure or meaning. Given a vocabulary set V=\{v_{1},v_{2},\dots,v_{p}\} , we define a substitution function f_{L}:V\rightarrow V^{\prime} such that: \forall v_{i}\in V,\quad f_{L}(v_{i})=v^{\prime}_{i},\quad\text{where }\text{Syn}(v_{i})=v^{\prime}_{i}. Here, \text{Syn}(v) ensures v^{\prime} is a valid synonym or equivalent phrase. This tests whether models are robust to surface-level changes. 3 Generating