Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning

Michael Hersche, Giacomo Camposampiero, Roger Wattenhofer, Abu Sebastian, Abbas Rahimi · 2024

Read Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning by Michael Hersche, Giacomo Camposampiero, Roger Wattenhofer, Abu Sebastian, Abbas Rahimi (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-05586.txt.

Introduction

Michael Hersche and co-authors pit big language models against a neuro-symbolic learner on Raven's matrices, in a 2024 paper. The classic visual puzzles hide arithmetic rules, progressions and sums, behind shapes. To isolate reasoning from eyesight, the team handed models the shapes' attributes as text, a perfect perception oracle.

Even so, GPT-4 and a seventy-billion-parameter Llama could not ace the center constellation of the I-RAVEN set. The autopsy blames arithmetic: the models fumble executing the very rules they can state. A neuro-symbolic rival encoding concepts as vectors, where adding vectors adds values, scored nearly perfectly. The contrast suggests fluent pattern talk is not the same as doing sums, and hybrid designs may bridge the gap.

Catalog record

CREATORMichael Hersche, Giacomo Camposampiero, Roger Wattenhofer, Abu Sebastian, Abbas Rahimi
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE38 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-05586

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Towards Learning to Reason: Comparing LLMs with Neuro-Symbolic on Arithmetic Relations in Abstract Reasoning Michael Hersche, Giacomo Camposampiero, Roger Wattenhofer, Abu Sebastian, Abbas Rahimi Abstract This work compares large language models (LLMs) and neuro-symbolic approaches in solving Raven’s progressive matrices (RPM), a visual abstract reasoning test that involves the understanding of mathematical rules such as progression or arithmetic addition. Providing the visual attributes directly as textual prompts, which assumes an oracle visual perception module, allows us to measure the model’s abstract reasoning capability in isolation. Despite providing such compositionally structured representations from the oracle visual perception and advanced prompting techniques, both GPT-4 and Llama-3 70B cannot achieve perfect accuracy on the center constellation of the I-RAVEN dataset. Our analysis reveals that the root cause lies in the LLM’s weakness in understanding and executing arithmetic rules. As a potential remedy, we analyze the Abductive Rule Learner with Context-awareness (ARLC), a neuro-symbolic approach that learns to reason with vector-symbolic architectures (VSAs). Here, concepts are represented with distributed vectors s.t. dot products between encoded vectors define a similarity kernel, and simple element-wise operations on the vectors perform addition/subtraction on the encoded values. We find that ARLC achieves almost perfect accuracy on the center constellation of I-RAVEN, demonstrating a high fidelity in arithmetic rules. To stress the length generalization capabilities of the models, we extend the RPM tests to larger matrices (3 × \times 10 instead of typical 3 × \times 3) and larger dynamic ranges of the attribute values (from 10 up to 1000). We find that the LLM’s accuracy of solving arithmetic rules drops to sub-10%, especially as the dynamic range expands, while ARLC can maintain a high accuracy due to emulating symbolic computations on top of properly distributed representations. Our code is available at https://github.com/IBM/raven-large-language-models . I Introduction Abstract reasoning is often regarded as a core feature of human intelligence. This cognitive process involves abstracting rules from observed patterns in a source domain, and applying them in an unseen target domain. With the ultimate aim to achieve human-level intelligence, abstract reasoning tasks have sparked the interest of many in machine learning research. Thanks to the availability of large datasets barrett_measuring_2018 ; zhang_raven_2019 ; hu_stratified_2021 , various learning-based methods, ranging from pure connectionist benny_scale-localized_2021 ; wu_scattering_2020 to neuro-symbolic zhang_abstract_2021 ; hersche_neuro-vector-symbolic_2023 ; hersche_probabilistic_2024 ; camposampiero_towards_2024 approaches, achieved promising results in this domain. More recently, the zero- and few-shot capabilities of large language models (LLMs) and their multi-modal variants have been tested on various abstract reasoning tasks such as verbal webb_emergent_2023 ; stevenson_large_2023 ; gendron_large_2024-1 ; lewis_evaluating_2024 or visual cao_what_2024 ; webb_emergent_2023 ; hu_-context_2023 ; mitchell_comparing_2024 ; camposampiero_abstract_2023 ; jiang_marvel_2024 ; ahrabian_curious_2024 ; zhang_how_2024 ; wust_bongard_2024 ; latif_systematic_2024_short ; lewis_evaluating_2024 analogies. One natural approach towards zero-shot visual abstract reasoning is to leverage multi-modal LLM ’s vision capabilities to solve the task end-to-end. However, these multi-modal models perform significantly worse than their text-only version mitchell_comparing_2024 , which might stem from a missing fine-grained compositional feature comprehension cao_what_2024 . As an additional help, LLMs have been provided with text-only inputs by giving them access to an oracle perception, i.e., providing perfectly disentangled representations webb_emergent_2023 ; hu_-context_2023 . While this generally improves their reasoning abilities, LLMs still fail to achieve perfect accuracy on many simple tasks. One example is represented by Raven’s progressive matrices (RPMs) raven_ravens_1938 , a benchmark that tests visual abstract reasoning capabilities by measuring the fluid intelligence of humans. Here, the state-of-the-art (SOTA) LLM -based approach hu_-context_2023 achieves only 86.4% accuracy in the center constellation of I-RAVEN hu_stratified_2021 , which we observe to be a gate-keeper for this task (see Section II.1 ). In contrast, recent neuro-symbolic approaches showed not only almost perfect accuracy on the center constellation of I-RAVEN, but also demonstrated high fidelity in out-of-distribution (OOD) settings. For instance, the Abductive Rule Learner with Context-awareness (ARLC) represents attribute values with high-dimensional, distributed representations based on vector-symbolic architectures (VSAs) plate_holographic_1995 ; plate_holographic_2003 ; gayler_vector_2003 ; kanerva_hyperdimensional_2009 . Learning the RPM rules boils down to a differentiable assignment problem of high-dimensional panel representations in a series of binding and unbinding operations, which can be solved with unconstrained optimization algorithms such as stochastic gradient descent (SGD). ARLC outperformed the SOTA LLM-based approach hu_-context_2023 both on in-distribution and OOD, thanks to relying on structured and similarity-preserving representations based on fractional power encoding (FPE) plate_holographic_2003 . Figure 1: This work compares the abstract reasoning capabilities of LLMs and neuro-symbolic ARLC on Raven’s progressive matrices (RPM) tests. a) An RPM example taken from the center constellation of I-RAVEN. The task is to find the empty panel at the bottom-right of the context matrix by selecting one of the answer candidates. b) Solving RPMs through LLM prompting. Visual attribute values are extracted from the I-RAVEN dataset and assembled to individual per-attribute text-only prompts. LLMs are prompted to predict the attribute of the empty panel. Finally, the attribute predictions are compared with the answer candidates, whereby the best-matching answer is selected as the final answer. c) Solving RPMs with neuro-symbolic ARLC that relies on distributed similarity-preserving representations and manipulates them via dimensionality-preserving operations; it learns rule-formulations as a differentiable assignment problem. This paper extends on the initial work on ARLC camposampiero_towards_2024 , by comparing its abstract reasoning capability with two prominent LLMs (GPT-4 openai_gpt-4_2024_short and Llama-3 70B dubey_llama_2024_short ) (see Fig. 1 ). Circumventing the perception by providing ground-truth attribute labels to the models allows us to measure their analogical and mathematical reasoning capabilities in isolation when such compositionally structured (i.e., disentangled) representations are provided. Our comprehensive prompting efforts lead to very high accuracy for Llama-3 70B (85.0%) and GPT-4 (93.2%), where the latter notably outperforms previous reports with GPT-3 hu_-context_2023 (86.4%) and GPT-4 o1-preview latif_systematic_2024_short (18.00%). This LLM ’s imperfect accuracy on the isolated task motivated us to further analyze their capability of detecting and executing different rules. In both GPT-4 and Llama-3 70B, we find a notable weakness in performing arithmetic rules that require row-wise additions or subtractions (e.g., see the last prompt in Fig. 2 ). To gain more insight about this behavior, we set up a new RPM dataset (I-RAVEN-X) that increases the grid size from 3 \times 3 to 3 \times 10, additionally allowing for a configurable dynamic range for the arithmetic computations. Also here, we observe a notable weakness in the arithmetic rule that gets even amplified by an increasing dynamic range. On the other hand, ARLC demonstrates high accuracy on larger grid sizes and allows to increase the dynamic range without further retraining, thanks to the the capability of adjusting the underlying structured FPE representations. II Datasets II.1 I-RAVEN We test the models on the center constellation of I-RAVEN hu_stratified_2021 (see Fig. 1 ). The test consists of a 3 \times 3 context matrix and eight answer candidate panels. Each panel contains an object, characterized by different attributes (shape, size, and color). The relation between each attribute’s value in different panels is governed by a well-defined set of rules: constant , progression , arithmetic , and distribute three . The task is to infer the rule governing each attribute in the context matrix and use it to determine the content of the missing (bottom-right) panel, selecting it within the eight candidate answers.

Read the complete text