Impact of Noise on LLM-Models Performance in Abstraction and Reasoning Corpus (ARC) Tasks with Model Temperature Considerations

Nikhil Khandalkar, Krishna Shinde, Pavan Yadav, Lokesh B. Ramegowda, Rajarshi Das · 2025

Read Impact of Noise on LLM-Models Performance in Abstraction and Reasoning Corpus (ARC) Tasks with Model Temperature Considerations by Nikhil Khandalkar, Krishna Shinde, Pavan Yadav, Lokesh B. Ramegowda, Rajarshi Das (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2504-15903.txt.

Introduction

Nikhil Khandalkar and co-authors test how noise and sampling heat sway language models on puzzle tasks in a 2025 study. The tasks, from a known reasoning corpus, ask for rules that map grids to grids on fresh cases. Strong scores under clean inputs may hide brittle skill.

One leading model solved all tasks with clean input, while two rival models solved none. Adding noise hurt every model, whatever its design or heat setting. The shared slide points to pattern matching that cracks under small shifts. The authors urge work on sturdier reasoning before such tools face messy real-world inputs.

Catalog record

CREATORNikhil Khandalkar, Krishna Shinde, Pavan Yadav, Lokesh B. Ramegowda, Rajarshi Das
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE229 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2504-15903

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Impact of Noise on LLM-Models Performance in Abstraction and Reasoning Corpus (ARC) Tasks with Model Temperature Considerations Nikhil Khandalkar, Krishna Shinde, Pavan Yadav, Lokesh B. Ramegowda, Rajarshi Das Abstract Recent advancements in Large Language Models (LLMs) have sparked interest in their structured reasoning capabilities, particularly in abstraction and pattern recognition tasks. The Abstraction and Reasoning Corpus (ARC) benchmark serves as a key evaluation tool for assessing AI models’ ability to generalize and solve novel reasoning tasks. While GPT-4o successfully solves all ARC tasks at zero noise, models such as DeepSeek R1 and LLaMA 3.2 fail to solve any, raising questions about their abstraction and generalization capabilities beyond pattern matching. To investigate this further, we evaluate these models under varying noise levels and temperature settings. Our findings indicate that introducing noise significantly degrades performance across all models, underscoring their fragility under uncertain conditions. This suggests that while some models demonstrate reasoning abilities, they remain highly sensitive to input perturbations, limiting their robustness. By analyzing how different architectures handle noise and uncertainty, we provide insights into the limitations of current AI systems in structured reasoning. Our study highlights the need for more resilient AI models that can adapt to real-world complexity, informing future research on improving generalization, robustness, and alignment with human cognitive flexibility. 1 1 1 code: https://github.com/Enkefalos-Technologies/arc-challenge 1 Introduction As AI systems advance in solving complex reasoning tasks, evaluating their ability to generalize and align with human-like problem-solving strategies becomes crucial. The Abstraction and Reasoning Corpus (ARC) Challenge, introduced by Franois Chollet serves as a benchmark for assessing an AI model’s capacity to perform abstract reasoning, a skill fundamental to human intelligence. Unlike traditional machine learning tasks that rely heavily on pattern recognition over large datasets, the ARC Challenge emphasizes few-shot learning, generalization, and abstraction, requiring models to infer underlying rules from minimal examples [ 1 ] . The ARC Challenge consists of diverse problem-solving tasks that demand conceptual reasoning, pattern recognition, and rule abstraction skills that closely resemble human cognitive processing. Each task presents input-output examples demonstrating a transformation rule, and the AI model must deduce the correct rule to generalize to new, unseen test cases. Since human cognition excels at such tasks through structured representation and high-level abstractions, ARC provides an ideal testbed for evaluating the alignment between AI models and human-like reasoning mechanisms. Early approaches to solving ARC primarily relied on symbolic AI and program synthesis using manually defined heuristics to infer rules from example pairs [ 2 ] . However, these methods faced scalability issues, as they required handcrafted representations for each task. With the rise of deep learning, researchers explored the application of convolutional neural networks (CNNs), transformers, and large language models (LLMs) to ARC [ 8 ] . While some studies demonstrated moderate success using pretrained language models, they often relied on pattern recognition rather than true abstraction, failing to solve tasks that required extrapolation beyond their training distribution [ 3 ] . More recent research evaluated and GPT-4o on ARC showing improved performance but still highlighting significant limitations, particularly in handling noise and reasoning under uncertainty. A major challenge identified in prior work is the lack of robust generalization. Studies have shown that LLMs trained on large text corpora struggle to infer abstract rules in a structured reasoning setting. Additionally, research on model robustness suggests that introducing noise into input examples leads to significant performance drops, indicating that current models are highly sensitive to minor perturbations [ 4 ] . Building upon these findings, our study conducts a comprehensive evaluation of GPT-4o, and LLaMA 3.2 on ARC tasks under different noise levels and temperature settings. By analyzing model performance in these varied conditions, we aim to provide deeper insights into the role of structured reasoning in LLMs and their limitations in handling uncertainty. [ 7 ] . Our work extends prior research by systematically quantifying how noise affects reasoning performance and identifying the architectural differences that contribute to these limitations. To address these concerns, we conduct a comprehensive evaluation of state-of-the-art models, including GPT-4o. LLaMA 3.2, under varying noise levels and model temperature settings. 2 Motivation The Abstraction and Reasoning Corpus (ARC) benchmark serves as a fundamental testbed for evaluating an AI model’s ability to infer abstract patterns and solve problems requiring human-like reasoning. Unlike traditional machine learning benchmarks that rely on large-scale data-driven pattern recognition, ARC challenges models to generalize from a limited number of examples using conceptual reasoning [ 1 ] . This ability to generalize is a hallmark of human intelligence but remains a significant challenge for modern AI systems [ 8 ] . In real-world applications, AI systems frequently encounter noisy, ambiguous, or incomplete data. However, current models, including large language models (LLMs) and deep learning-based approaches, often struggle to maintain robust performance under such conditions. Their reliance on statistical correlations rather than genuine abstraction makes them vulnerable to disruptions caused by small perturbations. Recent studies have demonstrated that AI models, including GPT-4o and multimodal models, perform inconsistently when abstract reasoning is required. These findings highlight the need for more structured approaches that enhance conceptual understanding and generalization [ 5 ] . By systematically introducing noise into ARC tasks, we aim to evaluate the extent to which AI models can adapt to uncertainty and maintain reasoning accuracy. Research on ARC and related benchmarks suggests that human cognition is significantly more robust when handling abstraction and generalization compared to existing AI models [ 9 ] . This analysis not only highlights the weaknesses of existing approaches but also helps guide future advancements in AI architectures that are more resilient to real-world imperfections . The findings from this study will contribute to the broader goal of developing AI systems that can reason abstractly, much like humans, while also improving their robustness in dynamic environments. 2.1 The Importance of Studying Noise in ARC Tasks Human cognition is inherently resilient to small distortions in information, allowing us to infer missing details and adapt to uncertainty. In contrast, AI models often struggle with unexpected variations in data. The ARC dataset consists of grid-based tasks, where each grid is a rectangular grid (list of lists) of integers between 0 and 9, representing different patterns and structures. Grid sizes range from 1×1 to 30×30, presenting a diverse set of reasoning challenges. For our research, we specifically analyze 2-shot and 3-shot examples, where the model is given two, three input-output pairs as demonstrations before making predictions on a test case. However, despite selecting a fixed number of demonstrations (2-shot, 3-shot), our approach is designed to handle ARC tasks of any dimension, ensuring that our findings apply broadly across different task complexities. By introducing controlled noise in the input or output grids, we systematically evaluate how AI models maintain their reasoning ability under increasing levels of uncertainty. This allows us to quantify the extent to which GPT-4o can generalize beyond clean examples and whether it can infer logical rules even when data is perturbed. 2.2 Investigating the Impact of Model Temperature on Performance The temperature parameter in AI models plays a critical role in determining prediction variability. A lower model temperature setting encourages deterministic outputs, ensuring stability and consistency in responses. Conversely, increasing the model temperature introduces greater

Read the complete text