Benchmarking and Understanding Compositional Relational Reasoning of LLMs
Ruikang Ni, Da Xiao, Qingye Meng, Xiangyu Li, Shihui Zheng, Hongliang Liang · 2024
Read Benchmarking and Understanding Compositional Relational Reasoning of LLMs by Ruikang Ni, Da Xiao, Qingye Meng, Xiangyu Li, Shihui Zheng, Hongliang Liang (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-12841.txt.
Introduction
Ruikang Ni and colleagues build a benchmark to test whether large language models can do compositional relational reasoning, a hallmark of human intelligence. Whether transformer models can solve such tasks, and how, was unclear. Their synthetic benchmark, Generalized Associative Recall, unifies the essence of several mechanistic interpretability tasks into one framework.
Evaluation shows the benchmark is hard enough to expose a fundamental deficiency in current models' relational reasoning. It is also tractable enough for systematic interpretability study. Using attribution patching, the authors discover core circuits that one model reuses across different tasks, plus a set of vital attention heads. Interventions confirm those heads' correct functioning drives task performance. Most strikingly, two classes of heads encode the abstract notions of true and false for these tasks. They play a fundamental role across models and tasks. The dataset and code are public, inviting others to dissect how reasoning is implemented.
Catalog record
| CREATOR | Ruikang Ni, Da Xiao, Qingye Meng, Xiangyu Li, Shihui Zheng, Hongliang Liang |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 54 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-12841
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Benchmarking and Understanding Compositional Relational Reasoning of LLMs Ruikang Ni, Da Xiao, Qingye Meng, Xiangyu Li, Shihui Zheng, Hongliang Liang Abstract Compositional relational reasoning (CRR) is a hallmark of human intelligence, but we lack a clear understanding of whether and how existing transformer large language models (LLMs) can solve CRR tasks. To enable systematic exploration of the CRR capability of LLMs, we first propose a new synthetic benchmark called Generalized Associative Recall (GAR) by integrating and generalizing the essence of several tasks in mechanistic interpretability (MI) study in a unified framework. Evaluation shows that GAR is challenging enough for existing LLMs, revealing their fundamental deficiency in CRR. Meanwhile, it is easy enough for systematic MI study. Then, to understand how LLMs solve GAR tasks, we use attribution patching to discover the core circuits reused by Vicuna-33B across different tasks and a set of vital attention heads. Intervention experiments show that the correct functioning of these heads significantly impacts task performance. Especially, we identify two classes of heads whose activations represent the abstract notion of true and false in GAR tasks respectively. They play a fundamental role in CRR across various models and tasks. Dataset and code — https://github.com/Caiyun-AI/GAR Introduction Compositional relational reasoning (CRR), the ability to reason about multiple types of relations between different entities and combine them to draw conclusions or make predictions, is a hallmark of human intelligence. Transformer large language models (LLMs) have become the de facto backbone for foundation models due to their exceptional performance on various tasks. An important open question is whether and how existing LLMs can solve CRR tasks. Attempting to answer this question, a number of works study the CRR capabilities of LLMs using synthetic benchmarks ( Weston et al. 2016 ; Lake and Baroni 2018 ; Clark, Tafjord, and Richardson 2020 ) . Compared with real-world benchmarks, synthetic benchmarks offer precise control over the data creation process, helping to understand the strengths and weaknesses of models on targeted tasks. Thus, they are more suitable for benchmarking CRR, which occurs relatively rarely in real-world corpora. Besides, we also want to understand the underlying mechanism by which the models solve CRR using mechanistic interpretability (MI) ( Bereska and Gavves 2024 ) analysis. One line of work studies compositional or multi-step reasoning ( Dziri et al. 2024 ; Press et al. 2023 ; Yang et al. 2024 ; Allen-Zhu and Li 2023 ; Ye, Li, and Allen-Zhu 2024 ; Zhang et al. 2022 ; Sanford, Hsu, and Telgarsky 2024 ; Brinkmann et al. 2024 ; Thomm et al. 2024 ; Zhao and Zhang 2024 ) . They use synthetic tasks to reveal some fundamental limitations of LLMs on such tasks, manifested as either poor generalization when task complexities of training and test set differ ( Dziri et al. 2024 ) , or the compositionality gap ( Press et al. 2023 ) , i.e., how often models correctly answer all sub-problems but not generate the overall solution. In these tasks, difficulty is mainly controlled by the number of reasoning steps. When the number becomes large, the over-complex input and usually poor performance of existing LLMs, especially open source ones, make in-depth MI study infeasible. Several studies train specialized models from scratch on the proposed tasks, but the conclusions drawn from analyzing these models may not necessarily apply to existing LLMs. Moreover, the benchmarks used by most of these works have only a single dimension of variety, namely, the number of steps, which captures only an aspect of CRR. Another line of research uses synthetic tasks for MI study, e.g. associative recall (AR) ( Ba et al. 2016 ; Fu et al. 2023 ; Olsson et al. 2022 ) , knowledge recall (KR) ( Meng et al. 2022 ; Geva et al. 2023 ) , greater-than ( Hanna, Liu, and Variengien 2023 ) and indirect object identification (IOI) ( Wang et al. 2023 ) . Using techniques such as path patching ( Wang et al. 2023 ) and attribution patching ( Syed, Rager, and Conmy 2023 ; Hanna, Pezzelle, and Belinkov 2024 ) , some works discover the circuits - subgraph of the model’s computation graph consisting of attention heads and MLPs - responsible for solving the tasks. These works deepen our understanding of the general working mechanism of LLMs. For example, the induction head mechanism found by studying AR tasks underpins the in-context learning capability of Transformer LLMs ( Olsson et al. 2022 ) . And the IOI circuit is also generalizable to other tasks ( Merullo, Eickhoff, and Pavlick 2024a ) . While the tasks used in these works are suitable for MI study, they are usually too simple for mainstream LLMs (e.g., a model as small as GPT-2-117M can do AR and IOI tasks perfectly) to reveal any deficiency of LLMs in CRR. They also lack variety; most current MI research is done on a single task without any systematic study. In summary, the lack of a CRR benchmark with both appropriate difficulty and sufficient variety in existing work hinders systematic MI studies on the CRR capabilities of LLMs. 1 1 1 An extensive review of related work is in Appendix A. In this paper, we make the following contributions: • We propose a new synthetic benchmark called Generalized Associative Recall (GAR) by integrating and generalizing the essence of several tasks in MI, e.g. associative recall, knowledge recall, indirect object identification (IOI), in a unified framework. GAR consists of a set of automatically generated tasks with varying forms (e.g. affirmative/negative, generation/classification) and difficulties. It is challenging enough to stress the CRR capability of mainstream LLMs, meanwhile simple enough for systematic MI study. • We evaluate existing LLMs, e.g. open source Llama-2/3 7B-70B, close source GPT 3.5/4, on GAR to show that it is challenging for these LLMs despite appearing simple. Scaling helps but the compositionality gap increases, revealing fundamental deficiency of these LLMs in CRR. • To understand how LLMs solve GAR tasks, we use attribution patching to discover the core circuits reused by Vicuna-33B across different tasks, and a set of vital attention heads. Intervention experiments show that the correct functioning of these heads significantly impacts task performance. Especially, we identify two classes of heads whose activations represent the abstract notion of true and false in GAR tasks respectively. Experiments show that they play fundamental roles in CRR across various models and tasks. To our knowledge, it is the first time that such heads are identified and studied in real LLMs. Generalized Associative Recall Benchmark Basic Idea: From AR and KR to GAR Two well studied synthetic tasks in MI are AR and KR, which inspire GAR. In AR, given a sequence of key-value ( K - V ) pairs as context and one of the keys as query Q , the model must recall the value associated with the specified key as answer A , e.g. “ H 1 C 4 M 7 … C \rightarrow 4 ”. Compared with AR that requires the model to recall the information in context, KR requires the model to retrieve the factual knowledge stored in its parameters. Given a subject Q and a relation, the model must predict the corresponding attribute A , e.g.