Inductive Linguistic Reasoning with Large Language Models
Raghav Ramji, Keshav Ramji · 2024
Read Inductive Linguistic Reasoning with Large Language Models by Raghav Ramji, Keshav Ramji (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-17819.txt.
Introduction
Raghav Ramji and Keshav Ramji test in 2024 how well large language models reason through tongue puzzles from rare languages. Such puzzles ask for rules to be read off a few seed pairs. They test both step logic and leap logic in a pure form.
Their two step plan first has a model dream up kin seed pairs, then uses them with the real seeds in the prompt. On one puzzle set, top model scores rose by up to eight points next to chain thought. Gains held when weak models made the kin seeds. A second games set showed broad gains as well.
Catalog record
| CREATOR | Raghav Ramji, Keshav Ramji |
|---|---|
| YEAR | 2024 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 75 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2412-17819
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inductive Linguistic Reasoning with Large Language Models Raghav Ramji, Keshav Ramji Abstract Evaluating large language models (LLMs) on their linguistic reasoning capabilities is an important task to understand the gaps in their skills that may surface during large-scale adoption. In this work, we investigate the abilities of such models to perform abstract multilingual reasoning through the lens of linguistic puzzles on extremely low-resource languages. As these translation tasks involve inductive and deductive reasoning from reference instances, we examine whether diverse auxiliary demonstrations can be automatically induced from seed exemplars, through analogical prompting. We employ a two-stage procedure, first generating analogical exemplars with a language model, and then applying them in-context along with provided target language exemplars. Our results on the modeLing dataset show that analogical prompting is effective in eliciting models’ knowledge of language grammar similarities, boosting the performance of GPT-4o by as much as 8.1% and Llama-3.1-405B-Instruct by 5.9% over chain-of-thought approaches. These gains are attributable to the analogical demonstrations, both when self-generated as well as when produced by weaker multilingual models. Furthermore, we demonstrate that our method generalizes to other tasks present in Linguistics Olympiad competitions, achieving sizable improvements across all problem types and difficulty levels included in the LINGOLY dataset with GPT-4o. We also report several findings about interesting phenomena which drive linguistic reasoning performance, suggesting that such puzzles are a valuable benchmark for new reasoning methods. 1 Introduction As the capabilities of large language models (LLMs) continue to grow, it is necessary to develop ways of testing the boundaries of their ability to reason over a wide range of languages. Adapting language models to low-resource languages is challenging due to a lack of high-quality annotated data in the target language for supervised fine-tuning. This has led to zero-shot and few-shot transfer learning approaches being more commonly employed ( Zoph et al., 2016 ; Nguyen & Chiang, 2017 ; Lin et al., 2019 ) . However, given the emergence of the in-context learning phenomenon, we hypothesize that this behavior can be used to enable few-shot generalization to new languages at inference time . In this work, we explore the task of linguistic reasoning , using linguistics puzzles akin to the International Linguistics Olympiad (IOL). Notably, in these puzzles, the target language is often extremely low-resource or functionally extinct ( Bean et al., 2024 ) . While prior work has largely examined the effect of vanilla in-context learning with English-target and target-English exemplars, chain-of-thought prompting, and traditional neural machine translation methods ( Chi et al., 2024 ; Şahin et al., 2020 ) , we believe that generating auxiliary exemplars can guide the model to more effectively learn grammar similarities over a language family. We introduce an approach based on analogical prompting ( Yasunaga et al., 2024 ) , which uses strong language models to self-generate exemplars of relevant problems given the test instance and performs in-context learning conditioned on those demonstrations. In our setting, the knowledge retrieval-like nature of analogical prompting allows us to test models’ parametric understanding of language families, performing inference to solve with both sets of demonstrations. We evaluate our approach on the modeLing ( Chi et al., 2024 ) dataset, consisting of unseen IOL-style problems. We find that strong models such as GPT-4o and Llama-3.1-405B-Instruct can identify the language family, similar languages within said family, generate exemplars in those similar languages, and apply them to solve the test puzzle. Furthermore, while weak models do not benefit significantly from using strong model-generated exemplars, strong models improve from using exemplars produced by weaker yet specialized multilingual models (e.g. Aya-35B). Our findings show that the ability of the model to deduce and apply rules, following inductive learning from the exemplars, largely influences performance; while there still exist gaps relative to the reasoning of human experts. We suggest that the linguistic reasoning task presents a fertile ground for research on new language model reasoning methods, to uncover how the skills which drive logical thinking may be imbued to models. Figure 1 : An illustration of our 2-stage analogical prompting approach, translating a phrase in Montenegrin to English. While prior works would solely provide exemplars translating between the source language and English and perform in-context learning, our method seeks diverse exemplars. Model M_{1} first identifies the language family (Slavic) and higher-resource languages in the family which the model has knowledge of (Croatian), then produces exemplars in those languages. Finally, both the original and generated set of exemplars are passed with the test puzzle to model M_{2} to perform the translation. M_{1}=M_{2} yields the self-generated analogical reasoning setting. 2 Analogical Prompting for Linguistic Reasoning Analogical prompting ( Yasunaga et al., 2024 ) avoids the need for annotated exemplars by relying on a strong model to generate exemplars which are related to the test instance, but are sufficiently diverse relative to one another and the test sample. Our approach of applying analogical prompts follows the human system 2 thinking framework of slow, deliberate reasoning ( Kahneman, 2011 ) . In chain-of-thought prompting for these puzzles, the model performs in-context learning with the given exemplars, learning the rules governing the language by induction, including the meaning of particular words, and using deduction to apply these rules to the test sample. This approach is supported by prior works demonstrating the ability of LMs to learn rules and attempt to apply them ( Qiu et al., 2024 ; Zhu et al., 2024 ) . Furthermore, we do not have access to complete grounding sources of human-written rules governing these low-resource languages, so we must rely on the LM to identify and generate these rules itself. However, as we expect the model to have little to no prior knowledge about the target language 1 1 1 We design our experiments to avoid leakage, but do not directly analyze test set contamination aside from zero-shot baselines. , we seek to leverage other languages with similar grammar structure which the model has learned in order to guide the language model’s reasoning process. We use language families as a taxonomically-grounded means of identifying similar languages as the target. The generated exemplars provide a source of reasoning support to the model, enabling it to perform inductive reasoning first in a cross-lingual manner over the diverse exemplars, and then deduce from its shared understanding. In Figure 1, given the test instance and the provided exemplars in Montenegrin, we leverage the model to (i.) identify the family of Montenegrin (Slavic Language Family), (ii.) select a few languages in the Slavic Language Family, and (iii.) generate example puzzles with their solutions in those selected languages, e.g. Croatian. Then, the provided exemplars and the generated cross-lingual demonstrations are given to the model, to solve the test puzzle. We desire for the model to produce exemplars from languages it has learned, in accordance with its pre-training distribution. In ( Qiu et al., 2024 ) , models have been shown to improve with more familiar exemplars (based on inclusion in the pre-training data). Furthermore, their work suggested that noisy demonstrations hurt performance; as the provided examples from the unseen target language could be considered as "noisy", we hypothesize that the generated exemplars can help to compensate. Exemplar Correctness. While one would ideally prefer to have