Divergence-based Safety Measure for Large Language Models via Rational Inattention

Anh Tung Nguyen, Quanyan Zhu · 2026

Read Divergence-based Safety Measure for Large Language Models via Rational Inattention by Anh Tung Nguyen, Quanyan Zhu (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2606-29081.txt.

Introduction

Nguyen and Zhu propose a safety measure for language models facing attacks hidden inside input embeddings, the numeric vectors representing text. Their measure is the worst-case divergence, a statistical distance, between the output distributions of the clean model and the attacked model, subject to a stealthiness limit on the attack. The limit rests on a neat equivalence: transformer attention behaves like rational inattention, an economics model of costly focus.

The authors analyse perfectly undetectable attacks within this framework and derive an upper bound on the measure through a Bregman-divergence argument. They then apply the measure to GPT-2 and GPT-Neo-125M. Both showed non-trivial output shifts under attack, and the measure separated their safety profiles. The contribution is a principled yardstick: it quantifies how far an attacker can move a model's behaviour while staying hidden.

Catalog record

CREATORAnh Tung Nguyen, Quanyan Zhu
YEAR2026
COLLECTIONPAPERS
FORMATTXT
FILE SIZE35 KB
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2606.29081

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Divergence-based Safety Measure for Large Language Models via Rational Inattention Anh Tung Nguyen, Quanyan Zhu Abstract This paper proposes a divergence-based safety measure for large language models (LLMs) under embedding-input attacks. The proposed measure quantifies the worst-case Kullback–Leibler divergence between the clean and attacked LLMs’ output distributions, subject to a stealthiness constraint. This constraint is constructed by leveraging the equivalence between transformer attention used in LLMs and rational inattention modeling human decision-making. We analyze the proposed divergence-based safety measure by investigating perfectly undetectable attacks and deriving its upper bound through a Bregman-divergence argument. The proposed safety measure is applied to two pretrained causal language models, GPT-2 and GPT-Neo-125M, to show nontrivial output-distribution shifts, illustrating that the measure can distinguish model-level safety profiles. 1 Introduction Large language models (LLMs), such as OpenAI GPT and Meta Llama, have been widely employed in many real-world applications thanks to their ability to generate text responses to human queries [ 11 ] . As LLMs are increasingly integrated into decision-support pipelines, human operators may in the near future consult LLM-generated outputs before making consequential decisions. This deployment paradigm introduces a system-level safety concern, where malicious perturbations of LLMs’ inputs may alter the model’s outputs and consequently influence the downstream decision made by a human operator. This concern motivates the need for quantitative safety measures for LLMs. More specifically, from a systems and control perspective, a measure quantifying the worst-case output perturbation caused by input attacks is surely demanded, e.g., [ 12 ] . Such a safety measure can support model comparison, runtime monitoring, and fine-tuning of LLM parameters to mitigate the impact of cyber-attacks. However, to the best of our knowledge, limited progress has been made from a systems and control perspective toward developing such safety measures for LLMs [ 3 ] . To fill this gap, this paper introduces a divergence-based safety measure for LLMs. We consider attacks on the embedding-input tokens and measure the attack impact by the Kullback–Leibler (KL) divergence between the clean and attacked LLMs’ output distributions. The proposed measure quantifies the worst-case output-distribution divergence induced by energy-bounded attacks that remain stealthy with respect to an internal monitoring output. A larger value of the proposed measure indicates that the LLMs’ output distribution can be changed more significantly by a stealthy attack. Hence, a smaller value corresponds to a safer model in the sense of reduced worst-case stealthy output impact. The internal monitoring output is constructed from transformer attention [ 14 ] , commonly used in LLMs, through a rational inattention (RI) perspective [ 6 ] . Recent evidence shows that commonly used LLMs, including OpenAI GPT and Meta Llama, may exhibit bounded-rational behavior similar to humans [ 7 ] . A possible explanation is that the softmax policy employed in scaled dot-product transformer attention [ 14 ] has the same functional form as the optimal action policy in the RI framework (see Figure 1 ). This connection is leveraged as a principled mechanism for constructing an internal monitoring output, which indicates the rationality level of LLMs’ outputs. The contributions of this paper are threefold. First, we show that the attention weights used by LLMs can be interpreted as the solution to an RI-type optimization problem. This establishes a structural correspondence between transformer attention mechanisms and rational-inattention models of decision-making. Second, this connection enables us to introduce a head-wise bounded-rationality measure. Third, we use this measure as a monitoring output in a safety problem where the embedding inputs are under attack. This yields a divergence-based safety measure that quantifies the worst-case KL output-distribution shift induced by energy-bounded attacks that remain stealthy with respect to the bounded-rationality monitor. We also investigate perfectly undetectable attacks and derive an upper bound on the stealthy attack impact. The proposed measure is illustrated by a controlled text-prediction experiment on two pretrained causal language models: GPT-2 [ 13 ] and GPT-Neo-125M [ 1 ] . The remainder of this paper is organized as follows. Section 2 revisits the rational-inattention framework and transformer attention, and establishes their structural connection through a shared softmax-based optimization form. We introduce the head-wise bounded-rationality measure, which is used to formulate the proposed divergence-based safety measure for LLMs under embedding-input attacks in Section 3. An analysis of the proposed safety measure is carried out in Section 4 by studying perfectly undetectable attacks and deriving an upper bound on the stealthy attack impact. We present a controlled candidate-prediction experiment comparing GPT-2 and GPT-Neo-125M under the same attack budget and trust tolerance in Section 5, while Section 6 concludes the paper. Figure 1: Illustration of the conceptual connection between RI in human decision-making and transformer attention in LLMs. In the RI framework, humans selectively allocate limited information-processing capacity to the most relevant observations before making decisions. Similarly, transformer attention mechanisms allocate attention weights to the most relevant context tokens when generating outputs. The figure highlights that both systems rely on the same softmax-based functional structure for selective information allocation, providing a bounded-rational interpretation of transformer attention in LLMs. This connection also enables us to leverage well-developed output perturbations for RI to quantify safety measures for LLMs. 2 Rational Inattention and Large Language Models In this section, we revisit the RI framework and the transformer attention commonly used in LLMs. The purpose is to establish a mathematical connection between them. 2.1 Rational inattention framework RI is a framework that formulates the human decision-making process when humans have a finite capacity for processing all the available data. In this framework, humans only observe a version of the true information via their customized lens, which can enhance the quality of the observed information by paying more attention at a higher cost. The better-quality data is expected to yield a higher utility for the human decision-maker. Therefore, the human decision-maker is interested in trading off the incremental utility and the extra information cost, which is the core formulation in the RI framework. We revisit the RI framework by denoting a latent state \omega\in\Omega , which is not directly observed and has a probability \pi(\omega) . On the other hand, humans can only observe a state s\in S , which has a policy \displaystyle\sigma:\Omega\to\Delta(S). (1) Here, \sigma(s|\omega) tells us the probability to observe s when \omega occurs. After observing s , we have an action policy \mu(a|s) over an action space {\mathcal{A}} , i.e., \displaystyle\mu:S\to\Delta({\mathcal{A}}). (2) With the action a and the latent state \omega , we obtain the corresponding utility u(a,\omega) . As discussed above, the information cost is formulated from the difference between observing s and the latent state \omega using the standard KL divergence as follows: \displaystyle I(\omega;s) \displaystyle\triangleq D_{\text{KL}}(\pi(\omega)\sigma(s|\omega)\|\pi(\omega)\sigma(s)) \displaystyle=\sum_{\omega\in\Omega,\,s\in{\mathcal{S}}}\pi(\omega)\sigma(s|\omega)\log\frac{\sigma(s|\omega)}{\sigma(s)}. (3) This information cost I(\omega;s) informs how well the state s is observed. This can also be interpreted as how much attention is paid to s when humans take an action, which is determined by solving the following optimization problem: \displaystyle\sup_{\sigma,\mu}~\sum_{\omega\in\Omega,s\in{\mathcal{S}},a\in{\mathcal{A}}}~\pi(\omega)\sigma(s|\omega)\mu(a|s)u(a,\omega)-\kappa I(\omega;s). (4) Here, the first term stands for the expected utility and the information cost is shown in the second term with \kappa>0 , which plays a key role in the trade-off between the utility and the information cost. We will discuss a solution to ( 4 ) in the following. Let us denote the latent-feedback

Read the complete text