Measuring Goal-Directedness

Matt MacDermott, James Fox, Francesco Belardinelli, Tom Everitt · 2024

Read Measuring Goal-Directedness by Matt MacDermott, James Fox, Francesco Belardinelli, Tom Everitt (2024) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2412-04758.txt.

Introduction

Matt MacDermott, James Fox, Francesco Belardinelli, and Tom Everitt present a 2024 paper that defines a measure of goal-directedness. Judging whether a system truly pursues a goal matters for AI safety and for the philosophy of agency. Their measure, called MEG, works inside causal models and step-by-step decision settings. It builds on a known frame from inverse reward learning, where goals are inferred from behavior.

MEG can score a system against one known goal, a set of candidate goals, or a set of outcome traits. The authors prove the measure meets several traits a good measure should have. They also give methods to compute it and test them on small cases. A shared yardstick could help labs rate how goal-driven an AI system is before it is trusted with open tasks.

Catalog record

CREATORMatt MacDermott, James Fox, Francesco Belardinelli, Tom Everitt
YEAR2024
COLLECTIONPAPERS
FORMATTXT
FILE SIZE44 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2412-04758

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

Measuring Goal-Directedness Matt MacDermott, James Fox, Francesco Belardinelli, Tom Everitt Abstract We define maximum entropy goal-directedness (MEG) , a formal measure of goal-directedness in causal models and Markov decision processes, and give algorithms for computing it. Measuring goal-directedness is important, as it is a critical element of many concerns about harm from AI. It is also of philosophical interest, as goal-directedness is a key aspect of agency. MEG is based on an adaptation of the maximum causal entropy framework used in inverse reinforcement learning. It can measure goal-directedness with respect to a known utility function, a hypothesis class of utility functions, or a set of random variables. We prove that MEG satisfies several desiderata and demonstrate our algorithms with small-scale experiments. 1 1 1 A notebook that can be used to apply our algorithms in your own tabular MDPs is available at https://colab.research.google.com/drive/1p_RA7EeN6DWx6x3M4v45R6L84koa4rf_ 1 Introduction In order to build more useful AI systems, a natural inclination is to try to make them more agentic . But while agents built from language models are touted as the next big advance [ Wang et al., 2024 ] , agentic systems have been identified as a potential source of individual [ Dennett, 2023 ] , systemic [ Chan et al., 2023 , Gabriel et al., 2024 ] , and catastrophic [ Ngo et al., 2022 ] risks. Agency is thus a key focus of behavioural evaluations [ Shevlane et al., 2023 , Phuong et al., 2024 ] and governance [ Shavit et al., 2023 ] . Some prominent researchers have even called for a shift towards designing explicitly non-agentic systems [ Dennett, 2017 , Bengio, 2023 ] . A critical aspect of agency is the ability to pursue goals. Indeed, the standard theory of agency defines agency as the capacity for intentional action – action that can be explained in terms of mental states such as goals [ Schlosser, 2019 ] . But when are we justified in ascribing such mental states? According to Daniel Dennett’s instrumentalist philosophy of mind [ 1989 ] , we are justified when doing so is useful for predicting a system’s behaviour. This paper’s key contribution is a method for formally measuring goal-directedness based on Dennett’s idea. Since pursuing goals is about having a particular causal effect on the environment. 2 2 2 This is a common but not universal view in philosophy, see Weirich [2020] . defining our measure in a causal model is natural. Causal models are general enough to encompass most frameworks popular among ML practitioners, such as single-decision prediction, classification, and regression tasks, as well as multi-decision (partially observable) Markov decision processes. They also offer enough structure to usefully model many ethics and safety problems [ Everitt et al., 2021a , Ward et al., 2024a , Richens et al., 2022 , Richens and Everitt, 2024 , Everitt et al., 2021b , Ward et al., 2024b , Halpern and Kleiman-Weiner, 2018 , Wachter et al., 2017 , Kusner et al., 2017 , Kenton et al., 2023 , Fox et al., 2023 , Richens and Everitt, 2024 , MacDermott et al., 2023 ] . Maximum entropy goal-directedness (MEG) operationalises goal-directedness as follows, illustrated by the subsequent running example. A variable D in a causal model is goal-directed with respect to a utility function \mathcal{U} to the extent that the conditional probability distribution of D is well-predicted by the hypothesis that D is optimising \mathcal{U} . (a) System of interest ➡ S D T (b) Model as causal Bayesian network ➡ S D T \bm{U}_{\theta} (c) Posit decision and utility variables ➡ S D T \bm{U}_{\theta} Predict? (d) Measure maximum predictive accuracy Figure 1 : Computing maximum entropy goal-directedness (MEG). 1 . A mouse begins at the centre of a gridworld ( Figure 1 a). It observes that a block of cheese is located either to the right or left ( S ) with equal probability, proceeds either away from it or towards it ( D ), and thus either obtains the cheese or does not ( T ). Suppose that the mouse moves left when the cheese is to the left and right when it is to the right, thus consistently obtaining the cheese. Intuitively, this behaviour seems goal-directed, but can we quantify how much? Figure 1 gives an overview of our procedure. We first model the system of interest as a causal Bayesian network ( Figure 1 b) with variables S for the cheese’s position, D for the mouse’s movement, and T for whether or not the mouse obtains the cheese. Then, we identify a candidate decision variable D and target variable T , hypothesising that the mouse optimises a utility function that depends on T ( Figure 1 c). Finally, we form a model of what behaviour we should expect from D if it is indeed optimising U and then measure how well this model predicts D ’s observed behaviour ( Figure 1(d) ). Ziebart [2010] ’s maximum causal entropy (MCE) framework suggests a promising way to construct a model for expected behaviour under a given utility function. However, there are several obstacles to applying it to our problem: it cannot measure the predictive usefulness of known utility functions, and it only finds the most predictive linear utility function. In practice, arbitrary known utility functions can be plugged in, but the results are not scale-invariant . We overcome these difficulties by returning to first principles and deriving an updated version of the MCE framework. In summary, our contributions are four-fold. We (i) adapt the MCE framework to derive maximum entropy goal-directedness (MEG), a philosophically-motivated measure of goal-directedness with respect to known utility functions, and show that it satisfies several key desiderata ( Section 3 ); (ii) we extend MEG to measure goal-directedness in cases without a known utility function ( Section 4 ); (iii) we provide algorithms for measuring MEG in MDPs ( Section 5 ); and (iv) we demonstrate the algorithms empirically ( Figure 3 ). Related Work. Inverse reinforcement learning (IRL) [ Ng and Russell, 2000 ] focuses on the question of which goal a system is optimising, whilst we are interested in to what extent the system can be seen as optimising a goal. We are not the first to ask this question. Some works take a Bayesian approach inspired by Dennett’s intentional stance. 3 3 3 After accepted publication, another related paper appeared [ Xu and Rivera, 2024 ] . Most closely related to our work is Orseau et al. [2018] , which applies Bayesian IRL in POMDPs using a Solomonoff prior over utility functions and an \varepsilon -greedy model of behaviour. This lets them infer a posterior probability distribution over whether an observed system is a (goal-directed) "agent" or "just a device". Our approach distinguishes itself from these by considering arbitrary variables in a causal model and deriving our behaviour model from the principle of maximum entropy. Moreover, since our algorithms can take advantage of differentiable classes of utility functions, our approach may be amenable to scaling up using deep neural networks. Oesterheld [2016] combines the intentional stance with Bayes’ theorem in cellular automata, but does not consider specific models of behaviour. Like us, Kenton et al. [2023] consider

Read the complete text