IMPROVING MULTIMODAL LLM’S ABILITY IN GEOMETRY PROBLEM SOLVING, REASONING, AND MULTISTEP SCORING

Avinash Anand, Raj Jaiswal, Abhishek Dharmadhikari, Atharva Marathe, Harsh Parimal Popat, Harshil Mital, Kritarth Prasad, Rajiv Ratn Shah, Roger Zimmermann · 2024

Read IMPROVING MULTIMODAL LLM’S ABILITY IN GEOMETRY PROBLEM SOLVING, REASONING, AND MULTISTEP SCORING by Avinash Anand, Raj Jaiswal, Abhishek Dharmadhikari, Atharva Marathe, Harsh Parimal Popat, Harshil Mital, Kritarth Prasad, Rajiv Ratn Shah, Roger Zimmermann (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-00846.txt.

Introduction

Anand and a large team present a paper from twenty twenty-four on a geometry dataset for vision-language models. The set gathers over two thousand problems from school textbooks. Augmentation grows it past five thousand geometry questions. Both numeric and proof-style problems are included. Each carries a step-by-step solution in one consistent format. Rival sets offer only final-answer style questions. Tests show open models still struggle at geometry reasoning.

Fine-tuning on the set lifts model skill clearly. Captioning and retrieval tricks are also weighed for gains. An automated grader compares full solutions to ground truth. The authors offer the set as a benchmark for geometric reasoning.

Catalog record

CREATORAvinash Anand, Raj Jaiswal, Abhishek Dharmadhikari, Atharva Marathe, Harsh Parimal Popat, Harshil Mital, Kritarth Prasad, Rajiv Ratn Shah, Roger Zimmermann
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE54 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-00846

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

IMPROVING MULTIMODAL LLM’S ABILITY IN GEOMETRY PROBLEM SOLVING, REASONING, AND MULTISTEP SCORING Avinash Anand, Raj Jaiswal, Abhishek Dharmadhikari, Atharva Marathe, Harsh Parimal Popat, Harshil Mital, Kritarth Prasad, Rajiv Ratn Shah, Roger Zimmermann Abstract . This paper presents GPSM4K, a comprehensive geometry multimodal dataset tailored to augment the problem-solving capabilities of Large Vision Language Models (LVLMs). GPSM4K encompasses 2157 multimodal question-answer pairs manually extracted from mathematics textbooks spanning grades 7-12 and is further augmented to 5340 problems, consisting of both numerical and theorem-proving questions. In contrast to PGPS9k, Geometry3K, and Geo170K which feature only objective-type questions, GPSM4K offers detailed step-by-step solutions in a consistent format, facilitating a comprehensive evaluation of problem-solving approaches. This dataset serves as an excellent benchmark for assessing the geometric reasoning capabilities of LVLMs. Evaluation of our test set shows that there is scope for improvement needed in open-source language models in geometry problem-solving. Finetuning on our training set increases the geometry problem-solving capabilities of models. Further, We also evaluate the effectiveness of techniques such as image captioning and Retrieval Augmentation generation (RAG) on model performance. We leveraged LLM to automate the task of final answer evaluation by providing ground truth and predicted solutions. This research will help to assess and improve the geometric reasoning capabilities of LVLMs. 1. Introduction Geometry problems, characterized by their intricate relationship between textual descriptions and visual representations, are exemplary benchmarks for evaluating multimodal numerical reasoning capabilities. Despite the longstanding recognition of automatic geometric problem-solving as a critical benchmark in AI research, the availability of suitable datasets is limited. This dearth stems from the inherent complexity and diverse array of information that geometry problems involve. Early on, there were small-scale datasets that relied heavily on manual annotation to assist geometric problem-solving models ( Seo et al., 2014 ) and GEOS ( Seo et al., 2015 ) is a dataset of SAT plane geometry questions with 186 questions. Then the Geometry3k ( Lu et al., 2021 ) was proposed, which not only increased the quantity of data but also enriched the geometric problem types in terms of geometric shapes and variable operators. At the same time, a larger and more diverse dataset, GeoQA ( Chen et al., 2021 ) , included clear annotations of the problem-solving process. It improved the universality and interpretability of multi-modal numerical reasoning. Language ability does not equal ’thinking’ or ’reasoning’ in LLMs. One of the long-term goals of artificial intelligence is to develop machines with the ability to reason mathematically. Existing geometry datasets like Geometry3K, PGPS9K lack a diverse range of problem statements, including Numerical Answer Questions and Theorem Proving Questions, essential for secondary-level education. To address this issue, one of the most direct and effective approaches is to enhance current Multimodal Large Language Models(MLLMs) by augmenting them with data containing high-quality descriptions of geometric information. However, the primary obstacle remains the small scale of existing geometric problem-solving datasets, typically offering only a few question-answer pairs. These datasets’ lack of geometric image descriptions and a narrow scope of problem-solving techniques limit models’ understanding of basic geometric concepts and their problem-solving abilities. To address the limitations of existing datasets, we develop a comprehensive multimodal geometry dataset named GPSM4K . GPSM4K consists of approximately 4,000 geometric image-caption pairs and question-answer pairs, extending the diversity and complexity of problems compared to Geometry3K. This paper investigates several key aspects of our research study • Improvement in LLM understanding: We examine how the integration of image captioning aids Large Language Models (LLMs) in comprehending images. The findings suggest that while image captioning can bridge some gaps between visual content and linguistic interpretation, its effectiveness depends on how advanced the captioning algorithms are and how relevant the captions are to the context as discussed in section. • Visual Model capabilities : The exploration into the Visual Encoder model’s interaction with Large Language Models (LLMs) reveals that, while the Visual Encoder is pivotal for processing visual data, it may inadvertently constrain the LLM’s capability if not adequately aligned with the LLM’s linguistic processing capabilities. This misalignment could limit the model’s overall efficacy in interpreting and solving geometry problems. • Step By Step Solution Framework and Diverse Question Types : Unlike traditional datasets, GPSM4K provides two solution versions for each problem: the textbook’s original solution and a reconstructed step-by-step version via Gemini Pro Vision. This structured approach facilitates a more detailed and methodical problem-solving process, allowing for clearer understanding and easier debugging of the solution path. Additionally, GPSM4K includes not only multiple-choice questions but also Numerical Answer Questions and Theorem Proving Questions. Our key contributions are • The introduction of GPSM4K, a diverse and extensive benchmark for geometry problem-solving, which includes question-answer pairs • A novel dataset structure featuring both original and detailed step-by-step solutions, alongside a variety of question types • Evaluation of image caption integration in multimodal geometry figures within GPSM4K to enhance contextual understanding of visual data and impact of Visual Encoder model in LLM’s ability. • Application of Retrieval-Augmented Generation (RAG) within GPSM4K to enrich multimodal understanding in geometry problem-solving, leveraging a vector database to retrieve relevant question-answer pairs and image descriptions during inference The remaining of this paper is organized as follows: Section 2 discusses the related works, including existing datasets and methods for geometry problem-solving in Large Vision Language Models (LVLMs). Section 3 presents the GPSM4K dataset, detailing the data collection process, structure, and augmentation techniques. Section 4 describes the experimental setup, including model training and evaluation methodologies. Section 5 showcases the results of our evaluations, highlighting the performance improvements achieved through various techniques such as image captioning and Retrieval-Augmented Generation (RAG). Section 6 provides an in-depth discussion of the implications of our findings, potential limitations, and future research directions. Finally, Section 7 concludes the paper by summarizing our contributions and key takeaways. Figure 1. Comparison of Geometry3K, PGPS9K, and our own GPSM4K datasets, showcasing geometric problems and their representations of complexity in solutions. 2. Related Works 2.1. Geometry Multimodal Datasets 2.1.1. PGPS9K PGPS9k is a comprehensive geometry problem dataset featuring 9,022 geometry problems accompanied by corresponding diagrams. Unlike prior datasets, PGPS9K uniquely provides both diagram annotations and solution programs ( Zhang et al., 2023 ) . Curated from five popular textbooks spanning grades 6-12, it comprises 30 problem types, covering a wide spectrum of plane geometry problems. Noteworthy properties include theorem-based problem-solving, a diagram-centric approach with over 90% dependency, abstract geometric representation, fine-grained variations in problem formulations, and condition redundancy mitigation. 2.1.2. Geoemtry3K Geometry3k is a large-scale geometry problem benchmark sourced from two prominent high school textbooks and online digital libraries ( Lu et al., 2021 ) . It encompasses challenging questions like irregular quadrilaterals and polygons and also presents questions with numerous unknown variables and operator types, often necessitating equation solving. Unlike some datasets, Geometry3k lacks annotated theorem application sequences due to resource-intensive annotation requirements. 2.1.3. Geo170K An innovative creation by Gao et al ( Gao et al., 2023 ) , leverages existing datasets through text-only LLMs to synthesize geometric visual-text data. This multi-modal dataset comprises approximately 60,000 geometric image-caption pairs and over 110,000 question-answer pairs, significantly surpassing its predecessors such as GeoQA++. Its expansive coverage and scale, 28 times larger than GeoQA++, enrich the landscape of geometric problem

Read the complete text