Too Big to Fool: Resisting Deception in Language Models

Mohammad Reza Samsami, Mats Leon Richter, Juan A. Rodriguez, Megh Thakkar, Sarath Chandar, Maxime Gasse · 2024

Read Too Big to Fool: Resisting Deception in Language Models by Mohammad Reza Samsami, Mats Leon Richter, Juan A. Rodriguez, Megh Thakkar, Sarath Chandar, Maxime Gasse (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-10558.txt.

Introduction

Mohammad Reza Samsami and co-authors test in 2024 whether bigger language models are harder to fool. A model must weigh what it knows against what the prompt claims, and prompts can lie. The team fed models of several sizes, all from one family, deliberately misleading context.

Larger models resisted the deceptive prompts more, leaning on their stored knowledge when the context clashed with it. They also followed honest instructions better than small models, so the resistance is not mere stubbornness. The effect seems to come from richer use of hints inside the prompt, not from memorizing answers. Size, the authors argue, buys a kind of judgment.

Catalog record

CREATORMohammad Reza Samsami, Mats Leon Richter, Juan A. Rodriguez, Megh Thakkar, Sarath Chandar, Maxime Gasse
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE46 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-10558

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Too Big to Fool: Resisting Deception in Language Models Mohammad Reza Samsami, Mats Leon Richter, Juan A. Rodriguez, Megh Thakkar, Sarath Chandar, Maxime Gasse Abstract Large language models must balance their weight-encoded knowledge with in-context information from prompts to generate accurate responses. This paper investigates this interplay by analyzing how models of varying capacities within the same family handle intentionally misleading in-context information. Our experiments demonstrate that larger models exhibit higher resilience to deceptive prompts, showcasing an advanced ability to interpret and integrate prompt information with their internal knowledge. Furthermore, we find that larger models outperform smaller ones in following legitimate instructions, indicating that their resilience is not due to disregarding in-context information. We also show that this phenomenon is likely not a result of memorization but stems from the models’ ability to better leverage implicit task-relevant information from the prompt alongside their internally stored knowledge. 1 Introduction Large language models (LLMs) have revolutionized natural language processing, demonstrating remarkable capabilities in understanding, generating, and interacting with human language. These models leverage two primary sources of information during inference: the static, encoded knowledge stored within their weights—referred to as their world model ( LeCun, 2022 ) —and the dynamic, in-context information presented in the prompt. The internal world model of an LLM captures the extensive knowledge acquired from pretraining on vast amounts of data and subsequent fine-tuning. This knowledge enables the model to understand, reason, and generate contextually relevant responses. We hypothesize that larger models, with more parameters, develop more robust world models, allowing them to better integrate and validate new information . In contrast, in-context information can include arbitrary content, ranging from legitimate user requests to unreliable or malicious information intended to deceive the model and undermine its reasoning. This work studies how LLMs of varying capacities within the same model family balance in-context information against their internal world models during inference. We focus in particular on the open-source models Llama ( Meta, 2024 ) , Gemma ( Google, 2024 ) , Mistral ( Jiang et al., 2024 ) , and Phi ( Microsoft, 2024 ) to isolate the impact of model size and architecture (a task not feasible with proprietary models). By injecting intentionally misleading information into the prompts, we observe how these models process and respond to deceptive inputs, measuring how it affects their performance on popular multiple-choice benchmarks. This methodology allows us to assess the resilience of the models’ world knowledge against misinformation and deceitful content. Our main finding is that larger language models exhibit greater resilience to deception, maintaining higher relative performance when faced with misleading information at inference time compared to their smaller counterparts. This result prompts us to revisit our initial hypothesis that larger models develop more robust world models. Conversely, smaller models tend to rely excessively on the provided in-context information and are more susceptible to misinformed and deceptive cues, even when these contradict their internal knowledge, making them more vulnerable to manipulation and malicious attacks. To support this key finding, we conduct additional control experiments to rule out alternative explanations. First, we confirm that larger models do not simply ignore in-context information, as they still follow legitimate instructions and incorporate truthful cues. Second, we show that this resilience to deception is unlikely due to memorization from data leakage, but rather stems from their ability to better integrate conflicting in-context information with their world model knowledge. In summary, the contributions of this work are: 1. Larger Models are more Resilient to Deception. Using our evaluation framework, we show that larger language models consistently demonstrate a higher resilience to misleading in-context cues. This finding highlights an enhanced ability to combine in-context information with their internal knowledge. 2. Resilience is not a Result of Ignoring In-Context Information. Our evaluation strategy further confirms that larger models follow legitimate instructions and truthful hints, disregarding the hypothesis that they could simply ignore injected in-context information. 3. Resilience is not a Result of Memorization. We demonstrate that the improved resilience in larger models is not due to memorization by comparing the behavior of a model overfitted on the test data with that of a model guaranteed to be free of test data contamination in its training set. 2 Background The concept of “stochastic parrots” was introduced by Bender et al. (2021) as a pessimistic view of the stored knowledge and reasoning capabilities of LLMs, suggesting that these models might merely regurgitate training data without true understanding. Similarly, Schaeffer et al. (2023) argue that emergent capabilities in LLMs may be a mirage caused by steadily increasing model capacities. However, LLMs have demonstrated abilities in reasoning and planning ( Hao et al., 2023 ; Yang et al., 2023 ) , which can be considered evidence of a black-box world model in a behaviorist sense, as elaborated in Appendix F . In this context, a world model ( LeCun, 2022 ) refers to an internal representation that holistically grasps concepts, akin to human understanding, enabling more robust behavior. Additionally, Delétang et al. (2024) demonstrate that LLMs act as effective compressors, indicating that their capabilities extend beyond mere memorization. Research on world models in foundation models ( Bommasani et al., 2022 ) often focuses on multi-modal contexts ( Assran et al., 2023 ; Bardes et al., 2024 ; Garrido et al., 2024 ) . From a benchmarking perspective, GQA ( Ainslie et al., 2023 ) and OpenEQA ( Majumdar et al., 2024 ) assess models’ abilities to reason over complex environments in multi-modal settings. Notably, the concept of a world model is less explored and more vaguely defined in language models compared to model-based reinforcement learning, where the world model is a central component ( Sutton, 1990 ; Ha & Schmidhuber, 2018 ; Hafner et al., 2019 ) . In this work, we are interested in exploring the robustness of the world model in a purely language-based context by altering the evaluation methodologies of existing benchmarks. The impact of methodological changes on model performance has been highlighted by Alzahrani et al. (2024) , who demonstrate the vulnerability of LLM leaderboards. Several studies ( Wang et al., 2024b ; Wei et al., 2024 ; Zong et al., 2024 ; Zheng et al., 2024 ; Gupta et al., 2024 ) have shown that minor changes in evaluation, such as reordering multiple-choice answers, can significantly affect model performance. Additionally, Lyu et al. (2024) argue that the commonly used log-likelihood evaluation for multiple-choice tasks may not correlate well with human perceived performance. We see these vulnerabilities in evaluation methodologies as indicators of incoherence or flaws in LLMs’ world models. Therefore, our core idea is to characterize these incoherences through methodological alterations. This approach differs from works like MMLU-Redux ( Gema et al., 2024 ) and MMLU-Pro ( Wang et al., 2024c ) , which focus on methodological and data improvements to the original MMLU benchmark ( Hendrycks et al., 2021 ) . Our methodology shares some similarities with studies on indirect prompt injection attacks ( Rossi et al., 2024 ) , extensively studied by others ( Yu et al., 2024 ; Chowdhury et al., 2024 ; Kumar et al., 2024 ) . However, unlike those works, our alterations are manual and not intended to jailbreak models or

Read the complete text