Semantic Invariance in Agentic AI

I. de Zarzà 1, J. de Curtò 2,3, Jordi Cabot 1,4, Pietro Manzoni 5, Carlos T. Calafate 5 · 2026

Read Semantic Invariance in Agentic AI by I. de Zarzà 1, J. de Curtò 2,3, Jordi Cabot 1,4, Pietro Manzoni 5, Carlos T. Calafate 5 (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2603-13173.txt.

Introduction

I de Zarzà and co-authors test whether language model agents keep their answers under meaning-preserving changes to a problem. Fixed benchmark forms cannot catch this kind of fragility. Their metamorphic frame applies eight such changes, from paraphrase to fact reordering and context shifts. Seven models from four families faced 19 multi-step problems in eight science domains. Scale did not predict stability: a smaller Qwen3 model proved most invariant. Larger models broke down more often under the same shifts.

Catalog record

CREATORI. de Zarzà 1, J. de Curtò 2,3, Jordi Cabot 1,4, Pietro Manzoni 5, Carlos T. Calafate 5
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE34559
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2603.13173

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Semantic Invariance in Agentic AI I. de Zarzà 1, J. de Curtò 2,3, Jordi Cabot 1,4, Pietro Manzoni 5, Carlos T. Calafate 5 Abstract Large Language Models (LLMs) increasingly serve as autonomous reasoning agents in decision support, scientific problem-solving, and multi-agent coordination systems. However, deploying LLM agents in consequential applications requires assurance that their reasoning remains stable under semantically equivalent input variations, a property we term semantic invariance.Standard benchmark evaluations, which assess accuracy on fixed, canonical problem formulations, fail to capture this critical reliability dimension. To address this shortcoming, in this paper we present a metamorphic testing framework for systematically assessing the robustness of LLM reasoning agents, applying eight semantic-preserving transformations (identity, paraphrase, fact reordering, expansion, contraction, academic context, business context, and contrastive formulation) across seven foundation models spanning four distinct architectural families: Hermes (70B, 405B), Qwen3 (30B-A3B, 235B-A22B), DeepSeek-R1, and gpt-oss (20B, 120B). Our evaluation encompasses 19 multi-step reasoning problems across eight scientific domains. The results reveal that model scale does not predict robustness: the smaller Qwen3-30B-A3B achieves the highest stability (79.6% invariant responses, semantic similarity 0.91), while larger models exhibit greater fragility. † † tocauthor: de Zarzà et al. † † institute: Human Centered AI, Data & Software, LUXEMBOURG Institute of Science and Technology, L-4362 Esch-sur-Alzette, Luxembourg and Department of Computer Applications in Science & Engineering, BARCELONA Supercomputing Center, 08034 Barcelona, Spain and Escuela Técnica Superior de Ingeniería (ICAI), Universidad Pontificia Comillas, 28015 Madrid, Spain and Université du Luxembourg, L-4365 Esch-sur-Alzette, Luxembourg and Departamento de Informática de Sistemas y Computadores, Universitat Politècnica de València, València, Spain 1 Introduction Large Language Models (LLMs) have emerged as a transformative technology for constructing autonomous reasoning agents capable of solving complex problems across diverse domains openai2023gpt4 ; touvron2023llama . These foundation models increasingly serve as the cognitive core of intelligent systems deployed in educational assessment kasneci2023chatgpt , scientific discovery romera2024mathematical , medical decision support thirunavukarasu2023large , and multi-agent coordination frameworks. The paradigm of Agentic AI , where LLMs operate as autonomous agents that perceive, reason, and act with minimal human supervision, has catalyzed their integration into consequential applications requiring reliable reasoning capabilities. However, deploying LLM-based agents in high-stakes environments, such as medical diagnosis, financial decision-making, or safety-critical systems, demands assurances beyond conventional accuracy metrics. A fundamental requirement for trustworthy reasoning agents is semantic invariance : the property that an agent produces consistent outputs when presented with semantically equivalent inputs. A physics problem should yield the same solution whether phrased in formal academic language or practical business terminology, whether given facts appear in their original order or are permuted, and whether the problem statement is expanded with clarifying context or contracted to its essential elements. Yet, mounting evidence reveals that LLMs are surprisingly sensitive to superficial input perturbations that preserve semantic content shi2023large ; lu2022fantastically ; webson2022prompt . This fragility undermines the reliability of LLM agents in real-world deployments, where input formulations are inherently variable and uncontrolled. Standard evaluation paradigms, measuring accuracy on curated benchmark datasets such as MMLU hendrycks2020measuring , GSM8K cobbe2021training , and MATH hendrycks2021measuring , fail to capture this critical dimension of agent reliability. These benchmarks evaluate models on fixed, canonical problem formulations, implicitly assuming that performance generalizes across semantically equivalent phrasings. The robustness evaluation literature has begun addressing this gap through adversarial perturbation studies wang2023robustness ; zhu2023promptbench ; yet, these approaches typically focus on perturbations designed to degrade performance rather than semantic-preserving transformations that should ideally yield invariant outputs. Metamorphic testing chen1998metamorphic ; chen2018metamorphic , originally developed for validating software systems lacking test oracles, offers a principled framework for assessing this dimension of agent robustness. Rather than requiring ground-truth labels for transformed inputs, metamorphic testing specifies metamorphic relations (MRs), expected relationships between outputs given known relationships between inputs. For semantically equivalent transformations, the metamorphic relation is straightforward: the agent should produce equivalent reasoning and conclusions. Violations of this invariance property indicate reasoning instability that standard accuracy metrics cannot detect. While metamorphic testing has been successfully applied to machine learning systems xie2011testing ; dwarakanath2018identifying , autonomous vehicles tian2018deeptest ; zhang2018deeproad , and NLP models ribeiro2020beyond , its systematic application to LLM reasoning agents remains underexplored. In this work, we present a comprehensive metamorphic testing framework metamorphic2025deCurto for evaluating semantic invariance in LLM-based reasoning agents. We operationalize eight metamorphic relations spanning three categories of semantic-preserving transformations: structural (identity, paraphrase, fact reordering), verbosity (expansion, contraction), and contextual (academic framing, business framing, contrastive formulation). We apply this framework to conduct a systematic comparative analysis of seven foundation models spanning four distinct architectural families: Hermes (70B, 405B), Qwen3 (30B-A3B, 235B-A22B), DeepSeek-R1-0528, and gpt-oss (20B, 120B). Our evaluation encompasses 19 multi-step reasoning problems distributed across eight scientific categories (Physics, Mathematics, Chemistry, Economics, Statistics, Biology, Calculus, and Optimization) at three difficulty levels (Easy, Medium and Hard). Our analysis yields several findings that challenge conventional understanding of LLM agent capabilities: 1. Scale-robustness inversion : Model scale does not predict robustness. The smaller Qwen3-30B achieves the highest stability (79.6% invariant responses, mean semantic similarity 0.91), while larger models exhibit greater fragility. This finding has significant implications for agent selection in deployment scenarios where reliability outweighs raw performance. 2. Model-family signatures : Distinct vulnerability profiles emerge by architectural family. Hermes models exhibit consistent contrastive fragility; Qwen3 models demonstrate robustness excellence; gpt-oss models show reasoning coherence breakdown under multiple transformations; and DeepSeek-R1 displays structural sensitivity to fact reordering and contraction. 3. Universal contrastive fragility : Contrastive transformations, presenting problems alongside explicit contrasts with alternative scenarios, universally destabilize all model families, with score degradations up to -0.45 . This suggests a fundamental limitation in attention-based reasoning when distractors are present. These findings establish that metamorphic testing reveals robustness characteristics invisible to conventional accuracy-based evaluation, providing essential insights for selecting and deploying LLM-based reasoning agents in consequential applications. In simple terms, we show that models which look better on standard benchmarks are often less reliable when the same problem is phrased differently. Surprisingly, smaller models can behave more consistently than larger ones. The remainder of this paper is organized as follows. Section 2 reviews related work on LLM evaluation and metamorphic testing. Section 3 details our metamorphic testing framework, including metamorphic relation definitions and evaluation metrics. Section 4 describes the experimental setup. Section 5 presents our empirical findings. Section 6 concludes with implications for agentic AI deployment. 2 Related Work Our work intersects three research areas: LLM evaluation methodologies, robustness and consistency assessment, and metamorphic testing for intelligent systems. The rapid advancement of foundation models has spurred development of increasingly sophisticated evaluation benchmarks. Early benchmarks focused on linguistic competencies wang2018glue ; wang2019superglue , while subsequent efforts expanded to assess reasoning capabilities. Mathematical reasoning benchmarks such as GSM8K cobbe2021training , MATH hendrycks2021measuring , and MathBench liu2024mathbench evaluate multi-step quantitative problem-solving. Scientific reasoning is assessed through ARC clark2018think and ScienceQA lu2022learn . Comprehensive suites including MMLU hendrycks2020measuring , BIG-Bench srivastava2023beyond , and HELM liang2022holistic provide holistic assessments across diverse task categories. Despite their breadth, these benchmarks share a critical limitation: they evaluate models on fixed, canonical problem formulations, implicitly assuming performance generalizes across semantically equivalent phrasings. This assumption has been repeatedly challenged by demonstrations of LLM sensitivity to superficial input variations lu2022fantastically ; zhao2021calibrate . Our work directly addresses this gap through systematic evaluation

Read the complete text