The Strong, Weak and Benign Goodhart’s law. An independence-free and paradigm-agnostic formalisation

Adrien Majka, El-Mahdi El-Mhamdi · 2025

Read The Strong, Weak and Benign Goodhart’s law. An independence-free and paradigm-agnostic formalisation by Adrien Majka, El-Mahdi El-Mhamdi (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-23445.txt.

Introduction

Adrien Majka and El-Mahdi El-Mhamdi wrote this 2025 paper giving Goodhart's law a formal treatment with fewer crutches. The adage says a measure turned into a target stops measuring well. Machine learning optimizers push proxies hard, so the law matters more each year, yet earlier formal versions assumed independence and one learning style.

The authors drop both assumptions and let the proxy and the true goal be coupled. When both the goal and the gap between goal and proxy are light-tailed, coupling changes nothing: the classic effect stands. With a heavy-tailed gap, they exhibit a case where over-optimization bites at a rate tied to that heaviness. Strong, weak, and benign forms of the law are separated cleanly. The paper supplies math for a folk warning.

Catalog record

CREATORAdrien Majka, El-Mahdi El-Mhamdi
YEAR2025
COLLECTIONPAPERS
FORMATTXT
FILE SIZE99 KB
VERSION1.0
LICENSEhttp://creativecommons.org/licenses/by/4.0/
RIGHTSCC BY
SOURCEarXiv

Source: https://arxiv.org/abs/2505-23445

Browse the full library · About the Tower of Babel

Full text

The opening of the work, from the library's hosted text:

The Strong, Weak and Benign Goodhart’s law. An independence-free and paradigm-agnostic formalisation Adrien Majka, El-Mahdi El-Mhamdi Abstract Goodhart’s law is a famous adage in policy-making that states that “When a measure becomes a target, it ceases to be a good measure”. As machine learning models and the optimisation capacity to train them grow, growing empirical evidence reinforced the belief in the validity of this law without however being formalised. Recently, a few attempts were made to formalise Goodhart’s law, either by categorising variants of it, or by looking at how optimising a proxy metric affects the optimisation of an intended goal . In this work, we alleviate the simplifying independence assumption, made in previous works, and the assumption on the learning paradigm made in most of them, to study the effect of the coupling between the proxy metric and the intended goal on Goodhart’s law. Our results show that in the case of light tailed goal and light tailed discrepancy, dependence does not change the nature of Goodhart’s effect. However, in the light tailed goal and heavy tailed discrepancy case, we exhibit an example where over-optimisation occurs at a rate inversely proportional to the heavy tailedness of the discrepancy between the goal and the metric. 1 Introduction From Charles Goodhart’s remark in the context of monetary economics [ 12 ] to its reformulation by Keith Hoskin [ 15 ] and its popularisation by Marylin Strathern [ 26 ] , Goodhart’s law remained unformalised. However with the growing place of metrics, a growing body of work expresses the need for a better understanding of how Goodhart’s law affects the reliance on metrics. To cite a few example : the AI governance literature [ 7 , 3 , 28 ] , fairness literature [ 30 ] , alignment research [ 16 , 28 ] . As a response, several attempts has been made to formalise Goodhart’s law to devise loss that can be provably resilient to it. Our work build on the work of [ 9 ] by taking the same formalisation but alleviating the independence hypothesis of their results. Our key contributions are the following : • We alleviate the independence hypothesis of previous work. • We derive a general result independent from the dependence structure with a general tail structure. • We conduct a detailed analysis in two cases of dependence which highlight the importance of coupling. • We propose a formal characterisation of different Goodhart’s law effect that captures the different strenght of Goodhart’s law. • For reproducibility purposes, we verified the computations in our proofs with the symbolic computation language Sympy [ 24 ] and provide the code to do so in the supplementary materials 1 1 1 LLMs are stricly not involved in any part of this paper . Paper structure. The rest of this paper is organised as follows. In Section 2 , we discuss some of the other work that formalised Goodhart’s law. Section 3 show our formal setup, in Section 4 we first provide a comprehensive overview of our results, followed by their formal statements and sketches of proofs (detailed proofs are available in the Appendix), finally, Section 5 discuss our results and proposes a comprehensive research agenda for future work. 2 Related Work Formalisations on Reinforcement Learning. RL was naturally a favourite setting to formalise Goodhart-Strathern’s law as the most notable cases of reward-hacking in AI appeared in reinforcement learning (RL) settings [ 4 , 1 , 13 ] . The first part of [ 18 ] shows the inefficiency of KL divergence to prevent reward hacking by showing that with heavy-tailed policy reward, a policy with arbitrarily high proxy reward but low penalty and low true reward always exists. [ 17 ] provide geometric explanation of Goodhart’s law in reinforcement learning while devising provably Goodhart-resilient early stopping criterion. [ 25 ] introduces a formalisation of reward hacking in the context of policy learning in RL. It proves several results on the general existence of reward hacking policy with respect to different reward function on different sets of policies. Formalisations on Supervised Learning. [ 14 ] intends to give a microfoundation to ML model to make them robust to the seminal critique of classic Keynesian models [ 20 ] . A regulator tries to make a prediction in a setting where, at test time, covariate can be manipulated by an agent to induce a more favorable decision from the regulator. Paradigm-Agnostic Formalisations. Two precedent works by D.Manheim and S.Garrabrant [ 22 , 23 ] provide interesting insight on general metrics design, by providing a towering view on metric potential flaw and set of case separation on Goodhart’s law respectively, while not fully formalising the problem. [ 31 ] devises a general framework for AI overoptimisation and draw results in the case of constrained ressources and partially specified goal. Their setup is inspired by incomplete contracting, but its reliance on state-space descriptions makes it more suitable to reinforcement learning. Previous works [ 9 , 18 ] provide a general formalisation of Goodhart’s law, where no assumption is made on the learning paradigm and show nuanced cases where Goodhart’s law holds or not depending on the relative thickness of the tail of the goal and the discrepancy. This was however done while assuming the discrepancy is independent from the goal. In our work, we follow the same paradigm-agnostic approach, but we alleviate the independence assumption. 3 Model We follow the same notation as in [ 9 ] and denote by G , M and \xi respectively the intended goal, the proxy metric being optimised, and the discrepancy between the goal and the proxy. Namely, an agent is optimising the function M as a proxy for G and G=M+\xi . Optimising is modelised in a mechanism agnostic way. Indeed, conditioning on M>m with m\rightarrow\infty model optimisation of the proxy metric. A basic “sanity check” for whether the proxy fails to capture the goal is to see if M and G remain correlated as we optimise M . When M and G are not correlated as M is being optimised, one can suspect that optimising M does not help make progress in the intended goal G , that is the weak Goodhart-Strathern case introduced in the formalism of [ 9 ] . When they keep being correlated, that is the no Goodhart-Strathern case. But looking at the correlation between G and M , as M is being optimised, is only a first check, one should focus on the intended goal’s behaviour. To do so, we also evaluate the expected value of the goal, as the proxy is being optimised, i.e., \lim_{m\to\overline{\mathfrak{S}_{M}}}\mathbb{E}\left[G|M>m\right] where for any random variable X , \mathfrak{S}_{X} is the support of it and \overline{\mathfrak{S}}_{X}=\sup\mathfrak{S}_{X} , \underline{\mathfrak{S}}_{X}=\inf\mathfrak{S}_{X} the superior and inferior limits of its support respectively. When \mathbb{E}\left[G|M>m\right] decreases as we optimise M that is the same strong Goodhart-Strathern case in [ 9 ] . In addition to the weak, strong and no Goodhart-Strathern, we introduce the benign Goodhart-Strathern . In the latter, M stops being correlated with G , but G keeps increasing as we optimise M . Qualitative definition Formal definition No Goodhart During optimisation the proxy stays informative and the goal G goes

Read the complete text