Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning Conference: Proceedings of the 2nd Workshop on Large Generative Models Meet Multimodal Applications ; October 28-November 1 2024; Melbourne, VIC, Australia Proceedings of the 2nd Workshop on Large G
Shihao Xu, Yiyang Luo, Wei Shi · 2024
Read Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning Conference: Proceedings of the 2nd Workshop on Large Generative Models Meet Multimodal Applications ; October 28-November 1 2024; Melbourne, VIC, Australia Proceedings of the 2nd Workshop on Large G by Shihao Xu, Yiyang Luo, Wei Shi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10455.txt.
Introduction
Shihao Xu, Yiyang Luo, and Wei Shi build in 2024 a multimodal model that solves geometry problems, diagrams included. Geometry strains language models because it mixes pictures with spatial reasoning, and data for solid geometry barely existed. The team scraped Chinese high school sites to build GeoMath, a set with worked answers for solid and plane problems.
Their model, Geo-LLaVA, trains with retrieval support and a meta-training stage, then uses in-context examples at answer time. It set the best scores on two benchmarks at the time, and it can describe solid figures and lay out reasoning steps, a first for this model class.
Catalog record
| CREATOR | Shihao Xu, Yiyang Luo, Wei Shi |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-10455
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Geo-LLaVA: A Large Multi-Modal Model for Solving Geometry Math Problems with Meta In-Context Learning Conference: Proceedings of the 2nd Workshop on Large Generative Models Meet Multimodal Applications ; October 28-November 1 2024; Melbourne, VIC, Australia Proceedings of the 2nd Workshop on Large Generative Models Meet Multimodal Applications (LGM3A ’24), October 28-November 1 2024, Melbourne, VIC, Australia DOI: 10.1145/3688866.3689124 ISBN: 979-8-4007-1193-0/24/10 CCS: Computing methodologies Knowledge representation and reasoning CCS: Computing methodologies Natural language processing CCS: Computing methodologies Computer vision representations Shihao Xu, Yiyang Luo, Wei Shi Abstract . Geometry mathematics problems pose significant challenges for large language models (LLMs) because they involve visual elements and spatial reasoning. Current methods primarily rely on symbolic character awareness to address these problems. Considering geometry problem solving is a relatively nascent field with limited suitable datasets and currently almost no work on solid geometry problem solving, we collect a geometry question-answer dataset by sourcing geometric data from Chinese high school education websites, referred to as GeoMath. It contains solid geometry questions and answers with accurate reasoning steps as compensation for existing plane geometry datasets. Additionally, we propose a Large Multi-modal Model (LMM) framework named Geo-LLaVA, which incorporates retrieval augmentation with supervised fine-tuning (SFT) in the training stage, called meta-training, and employs in-context learning (ICL) during inference to improve performance. Our fine-tuned model with ICL attains the state-of-the-art performance of 65.25% and 42.36% on selected questions of the GeoQA dataset and GeoMath dataset respectively with proper inference steps. Notably, our model initially endows the ability to solve solid geometry problems and supports the generation of reasonable solid geometry picture descriptions and problem-solving steps. Our research sets the stage for further exploration of LLMs in multi-modal math problem-solving, particularly in geometry math problems. 1. Introduction Figure 1 . Example of geometry question answering. LLaVA-1.5 often provides a vague image description, omitting many details and leading to wrong answers. GPT-4V struggles to understand geometry graphs, resulting in incorrect answers. In contrast, our Geo-LLaVA offers concise and accurate solutions to geometry problems. Image that illustrates the samples for solving geometry questions by LLMs. There has been an increased interest in using deep learning models, especially LMMs, for addressing computer vision challenges recently. Aligning image adapters with large language models has achieved remarkable success in image captioning and visual question answering (VQA), highlighting their powerful reasoning and visual understanding capabilities ( bordes_introduction_2024 ) . Despite their accomplishments in these areas, there has been limited investigation into utilizing LLMs for more complex multi-modal math problems, particularly geometry-related ones. The debate between symbolic and probabilistic approaches in mathematical reasoning persists. Traditionally, solving geometry problems involves analyzing diagrams and texts, converting them into logical expressions using formal symbolic language, and applying predefined geometry theorems to find solutions ( chen2021geoqa ; zhang2023multi ) . Alternatively, geometry problem-solving can be viewed as text generation with multi-modal input, which is more generalized and applicable to a broader range of mathematical problems, including trigonometry and vector graphics. By examining probabilistic approaches, we can better understand their strengths and limitations, leading to more effective strategies for solving geometry problems. Current Large Multi-modal Models (LMMs) have shown promising capabilities in visual understanding and question-answering tasks, as demonstrated by models such as BLIP-2 ( li2023blip ) , LLaVA ( llava ) , Flamingo ( alayrac2022flamingo ) , MiniGPT4 ( zhu2023minigpt ) , and InstructBLIP ( instructblip ) . However, these models still lack a deep comprehension of geometry images, which is crucial for solving geometry problems. Additionally, existing small language models often lack the mathematical reasoning abilities to solve complex math problems effectively. In this paper, we propose a solution to the challenges of multi-modal math problem-solving by introducing an LLM framework called Geo-LLaVA. The main contributions of this paper include: • We form a geometry question-answer dataset, GeoMath, with reasoning steps from Chinese high school education websites and expand the dataset by collecting more data from existing datasets such as GeoQA+ ( chen2021geoqa ) , Unigeo ( chen2022unigeo ) , and PGPS9K ( zhang2023multi ) , and creating reasoning steps for them. • We employ a new LLM framework named Geo-LLaVA with around 13 billion parameters. It can effectively generate reasoning steps and answers, which is the first model exploring both plane and solid geometry problems. • As far as we know, this is the first multimodal meta-training ( min2021metaicl ) method to enhance geometry problem-solving ability, which shows the state-of-the-art performance on multiple geometry datasets, including GeoQA+ and GeoMath. 2. Related Work Meta-Training Approaches The general issue of meta-training ( schmidhuber1987evolutionary ) , which encompasses few-shot learning, has been studied for numerous years. Recently, meta-training has emerged as a significant technique in machine learning. Using prior knowledge, it aims to create models that can swiftly adapt to new tasks with minimal data. meta-training is particularly pertinent in geometry problem-solving, where the diversity of problems can vary greatly. The most recent meta-training models, such as MAML ( finn2017model ) , Hypernetworks ( ha2022hypernetworks ) , and ( ravi2016optimization ) , have shown promise in rapidly adapting to new tasks with few-shot learning capabilities. Multi-Modal Large Language Model Concurrently, the success of LLMs has inspired investigations into vision-language interaction, resulting in the development of multi-modal large language models (MLLMs) ( llava ; li2023blip ; instructblip ; weng2022large ; 2023GPT4VisionSC ; bai2023qwen ) . These models have demonstrated remarkable abilities in generating detailed descriptions and engaging in dialogue based on visual inputs. Nonetheless, we observe that even the most advanced MLLMs struggle with resolving geometric problems using diagrams and figures. Geometry Problem Solving by LLMs Recently, math-specific LLMs such as Llemma ( azerbayev2023llemma ) and Mathcoder ( wang2023mathcoder ) have shown significant capabilities in text-only mathematical reasoning tasks and are competitive with general large language models like GPT-4 ( achiam2023gpt ) and PaLM-2 ( anil2023palm ) on a much smaller scale. Notably, AlphaGeometry ( Trinh2024AlphaGeo ) has exhibited impressive performance in solving challenging geometry problems, though it cannot process images and must rely on text descriptions. Current math-specific multimodal models, such as G-llava ( gao2023g ) , UniMath ( liang2023unimath ) , and UniChart ( masry2023unichart ) , are primarily focused on plane geometry or chart-based problems and still lag behind general multimodal models such as GPT-4V ( 2023GPT4VisionSC ) in benchmark testing. Yet, no works have utilized RAG or meta-training techniques on geometric problem-solving. 3. Method The Geo-LLaVA model, illustrated in Figure 1 , consists of a retrieval network and an LMM backbone. The retrieval network’s role is to fetch similar questions and their solutions as in-context samples during the training and inference phases. In the following section, we offer a comprehensive overview of the design process for our retrieval network. Next, we delve into our method of fine-tuning the model using image captioning, question-answering, and geometry math-solving datasets through Meta in-context learning. Lastly, we will describe how we have integrated a multi-modal chain of thoughts (CoT) during the inference phase to boost the model’s performance even further. Figure 2 . The overview of Geo-LLaVA model architecture. The training process includes three steps: training the retrieval model, augmenting data with the meta-training dataset, and fine-tuning the Language Model with the LoRA module. The question retrieval network is trained using inputs