AI Testing Should Account for Sophisticated Strategic Behaviour

Vojtěch Kovařík, Eric Olav Chen, Sami Petersen, Alexis Ghersengorin, Vincent Conitzer · 2025

Read AI Testing Should Account for Sophisticated Strategic Behaviour by Vojtěch Kovařík, Eric Olav Chen, Sami Petersen, Alexis Ghersengorin, Vincent Conitzer (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2508-14927.txt.

Introduction

Vojtěch Kovařík and co-authors argue that AI tests must grant the tested a mind for the test, in a stance paper from 2025. A strong tool may know it is in a test and play to the test, which makes the grade a tale of the test more than of the tool at work.

Two claims frame the piece. First, grades stay dear just if they book sly play by the tool. Next, game math can frame and stress the tales that safety cases spin from grades. Past tools, past work, and a clean toy case back the claims. Paths for new work close the piece: tests built for sly hands, they say, are the next wall.

Catalog record

CREATORVojtěch Kovařík, Eric Olav Chen, Sami Petersen, Alexis Ghersengorin, Vincent Conitzer
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE47 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2508-14927

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

AI Testing Should Account for Sophisticated Strategic Behaviour Vojtěch Kovařík, Eric Olav Chen, Sami Petersen, Alexis Ghersengorin, Vincent Conitzer Abstract This position paper argues for two claims regarding AI testing and evaluation. First, to remain informative about deployment behaviour, evaluations need account for the possibility that AI systems understand their circumstances and reason strategically. Second, game-theoretic analysis can inform evaluation design by formalising and scrutinising the reasoning in evaluation-based safety cases. Drawing on examples from existing AI systems, a review of relevant research, and formal strategic analysis of a stylised evaluation scenario, we present evidence for these claims and motivate several research directions. 1 Introduction Testing and evaluation are fundamental components of modern AI development. Before deployment, AI systems undergo capability assessments to verify their performance and testing on safety benchmarks, as well as other scrutiny such as red-teaming or interpretability analysis, to verify their safety. After deployment, they are often subject to continual monitoring and incident reporting. Throughout this paper, we use “testing” and “evaluation” interchangeably to refer to approaches that rely on examples of input-output behaviour to make inferences about the system. This includes approaches that treat AIs as black boxes, but also other methods such as interpretability. The key characteristic is that information about some critical properties of the system comes primarily from observed behaviour. We thus view testing-based approaches as distinct from other methods such as formal verification, mathematical proofs, or informal reasoning about the AI’s design. (While we focus on testing, we view the different methods as complementary and expect them to be used in conjunction.) Evaluations can serve multiple purposes. They can help with training the AI by providing feedback, or with preventing harmful behaviour by identifying unsuitable AI systems. They can also help build confidence in the AI’s safety, by forming the empirical backbone of safety cases —structured rationales arguing that an AI is unlikely to cause unacceptably harmful outcomes [ 8 , 13 ] . Furthermore, evaluation-based safety cases might, in the near future, become relevant to AI regulation [ 13 ] . Figure 1: An early instance of strategic behaviour enabled by situational awareness [ 6 ] . Claude 3.7 correctly recognises it is facing a test and decides against misbehaviour. While it does not seem to realise that its reasoning will be read, future AIs might not be similarly limited. However, all forms of evaluation rely on a crucial assumption: that the behaviour observed during testing is informative about the AI’s impact when deployed. This assumption, however, may break down when evaluation is applied to more advanced systems. One reason for this is that advanced systems might be capable of strategic behaviour . That is, AI systems that consistently and competently pursue objectives may tailor their strategies to account for the existence of the evaluation. For example, an AI trying to deceive potential evaluators might obfuscate its actions to decrease the chance that its misbehaviour is detected [ 7 ] or prioritise misbehaviour that could be attributed to an honest mistake. This problem could be exacerbated if the AI system in question possessed a particularly high level of situational awareness , i.e., a comprehensive understanding of its circumstances. Such a system might recognise when it is being evaluated and use this knowledge to modify its behaviour. This could manifest in at least two concerning ways. First, the system might act in accordance with the evaluator’s preferences during testing while pursuing different objectives during deployment (as in the Volkswagen emissions scandal, where diesel cars were programmed to detect tests and temporarily enter a low-emission mode [ 43 ] ). Second, it might strategically underperform during evaluation to avoid revealing its true capabilities, a dynamic known as sandbagging [ 74 ] . (Note that none of the discussion relies on the AI possessing consciousness, desires, true understanding, or other contentious properties. Rather, for this paper’s concerns to apply, it suffices that the AI behaves as if it were reasoning strategically—as in the Volkswagen example—regardless of how this arises mechanistically. For more discussion, see Footnote 1 .) This paper defends two points. First, we claim that (1) accounting for situational awareness and strategic behaviour should be a key concern in AI testing. Second, given that game theory provides a formal framework for analysing interactions between strategic agents, we argue that (2) game-theoretic analysis can play an important role in addressing this challenge, particularly by helping formalise and scrutinise the reasoning in evaluation-based safety cases . More precisely, (1) is meant to imply that, in contrast to the established practice in AI, future evaluations that do not account for the potentially sophisticated strategic nature of AIs are likely to become uninformative. (2) then proposes a vital addition to the toolbox used to study and design evaluations. The remainder of the paper develops these arguments and ties them to promising research directions. Section 2 starts by reviewing historical examples of evaluation failures in AI, examining the empirical evidence for deception and related behaviour in current AIs, and discussing theoretical arguments about AI development trajectories. Section 3 then considers a stylised example of evaluating a safety-critical AI system and uses it to demonstrate how game theory can be used to formalise, strengthen, or invalidate evaluation-based safety arguments. Beyond supporting claims (1-2), Sections 2 - 3 serve a second purpose: motivating and informing future research. They do this by illustrating key dynamics and concepts relevant to AI evaluation, while also reviewing existing work in this area. Since (research on) strategic behaviour in AI evaluation is still nascent, many important questions remain unexplored or only preliminarily investigated. As a result, even straightforward extensions or follow-ups to existing work could yield valuable insights. Building further on this point, Section 4 outlines several research directions that we consider particularly promising. These range from empirical investigation of current AI systems and improved evaluation methodologies to theoretical analyses and experimentation with evaluation design. We close by discussing alternatives to the position expressed ( Section 5 ) before concluding ( Section 6 ). 2 Importance of strategic behaviour and situational awareness We now review several types of evidence for claim (1), that although accounting for situational awareness and strategic behaviour might not have been a serious concern in AI testing thus far, it should be a key concern going forwards. 2.1 Historical examples of undermining evaluations There are many documented cases where AI played a crucial role in rendering evaluations uninformative. So far, however, the impulse for deception most often came from humans. A representative example is the already mentioned diesel emissions scandal, where a number of cars were discovered to contain software that could detect an ongoing emissions test and temporarily switch to a low-emissions mode [ 43 ] . Some further examples are: • Some smartphones use software that boosts performance during common benchmarks [ 29 ] . • In cybersecurity, viruses attempt to distinguish between honeypots (made by antivirus companies) and genuine targets, hoping to attack the latter while remaining undetected. Antivirus companies respond by making more sophisticated honeypots, resulting in an arms race [ 24 ] . • HP printers accepted third-party ink cartridges during testing, but later started rejecting them after a firmware update [ 1 ] .

Read the complete text