Subjective functions
Samuel J. Gershman · 2025
Read Subjective functions by Samuel J. Gershman (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2512-15948.txt.
Introduction
Samuel Gershman asks where objective functions come from in a 2025 paper on subjective functions. Human intelligence invents new goals on the fly, not just pursues given ones. A subjective function is a higher order objective defined relative to the agent's own features rather than an external task.
Expected prediction error serves as the concrete example: an agent can value states by how much they would reduce its own forecasting error. Gershman develops that example and traces connections to ideas in psychology, neuroscience, and machine learning. The proposal aims to explain how artificial systems might synthesize their own objectives the way people appear to do.
Catalog record
| CREATOR | Samuel J. Gershman |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 26 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2512-15948
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Subjective functions Samuel J. Gershman Abstract Where do objective functions come from? How do we select what goals to pursue? Human intelligence is adept at synthesizing new objective functions on the fly. How does this work, and can we endow artificial systems with the same ability? This paper proposes an approach to answering these questions, starting with the concept of a subjective function, a higher-order objective function that is endogenous to the agent (i.e., defined with respect to the agent’s features, rather than an external task). Expected prediction error is studied as a concrete example of a subjective function. This proposal has many connections to ideas in psychology, neuroscience, and machine learning. 1 Introduction Objective functions are central to all learning systems (both natural and artificial). The way we distinguish learning from other kinds of dynamics is the fact that learning produces (at least in expectation or asymptotically) an improvement in performance as measured by an objective function. 1 1 1 For example, passive wear and tear degrades the function of living organisms and robots over time, but this is not learning, because it cannot be understood in terms of performance improvement over time. Many different objective functions have been proposed, and it’s not clear that all intelligence can be subsumed by a single ‘‘ultimate’’ objective, such as reproductive fitness. 2 2 2 Even the reasonable argument that all forms of biological intelligence arose from natural selection is not very helpful for elucidating the underlying principles that give rise to intelligent behavior. Perhaps the problem is that the quest for a single objective function is misguided. An important characteristic of human-like intelligence may be the ability to synthesize objective functions . This only kicks the can down the road, of course. What principle disciplines the choice of objective function? Wouldn’t any such principle constitute a higher-order objective function? If so, then we would be back to where we started, the quest for a single objective function. A true subjective function (rather than a higher-order objective function) is endogenous to the agent. To understand what this means, consider a typical way to define an objective function: stipulate some reward or supervision signal, then score an agent based on how well it maximizes expected reward or minimizes expected error. These signals are exogenous to the agent in the sense that their definitions do not depend on any feature of the agent; they can be applied uniformly to any agent. In contrast, a subjective function is endogenous to the agent in the sense that the definition of the signal that the agent is optimizing depends on features of the agent. This note describes a subjective function that can be used to design an agent capable of open-ended learning. It then discusses how it connects to observations from psychology and neuroscience, as well as related ideas in machine learning. 2 Preliminaries We model the world as a Markov decision process (MDP) consisting of the following components: • A state space \mathcal{S} . • An action space \mathcal{A} . • A transition distribution T(s^{\prime}|s,a) . • The agent chooses actions according to a policy \pi(a|s) . Importantly, we do not assume a fixed reward function. Instead, we allow the agent to choose its own reward function. For concreteness, we will study the case where the reward function is parametrized by a specific goal state g\in\mathcal{S} : \displaystyle R_{g}(s)=\mathbb{I}[s=g]. (1) Thus, the reward is 1 only when the agent has reached the goal state. It’s relatively straightforward to extend this setup (e.g., to reward functions that are linear in some features space), but the goal-based framework is appealingly simple and applicable to many environments that are natural for humans. 3 Design principles Principle 1: agents select policies that maximize expected prediction error A standard assumption in reinforcement learning (RL) theory is that agents seek to maximize expected discounted future reward, or value : \displaystyle V^{\pi}_{g}(s) \displaystyle=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}R_{g}(s_{t})\bigg|s_{0}=s,\pi,g\right] \displaystyle=R_{g}(s)+\gamma\sum_{a}\pi(a|s)\sum_{s^{\prime}}T(s^{\prime}|s,a)V^{\pi}_{g}(s^{\prime}), (2) where t indexes time and \gamma\in[0,1) is a discount factor governing how the agent values long-term reward. The second equality is the Bellman equation. We instead adopt the assumption that the agent seeks to maximize expected prediction error (EPE), which replaces the reward with the temporal difference (TD) prediction error \delta_{t} : \displaystyle\pi^{\ast}_{g}(\cdot|s)=\operatornamewithlimits{argmax}_{\pi(\cdot|s)}U^{\pi}_{g}(s), (3) \displaystyle U^{\pi}_{g}(s)=\mathbb{E}\left[\sum_{t=0}^{\infty}\gamma^{t}\delta_{t}\bigg|s_{0}=s,\pi,g\right], (4) \displaystyle\delta_{t}=R_{g}(s_{t})+\gamma\hat{V}^{\pi}(s_{t+1})-\hat{V}^{\pi}(s_{t}), (5) where \hat{V}^{\pi}_{g} is the agent’s estimate of the value function. Intuitively, the EPE measures a form of goal progress, because \delta_{t}\approx\dot{V}^{\pi}_{g} prior to goal attainment (i.e., the prediction error approximates the rate of change in the goal-dependent value). Eq. 4 is a telescoping series (intermediate terms cancel out), allowing us to express it as follows: \displaystyle U^{\pi}_{g}(s)=V^{\pi}_{g}(s)-\hat{V}^{\pi}_{g}(s), (6) where we have assumed that \hat{V}^{\pi}_{g} is bounded so that \lim_{T\rightarrow\infty}\gamma^{T}\hat{V}^{\pi}_{g}(s_{T})=0 . Because the TD error is used to update value estimates, it is also necessary to assume that \hat{V}^{\pi}_{g} is frozen (or slowly changing) when computing \delta_{t} . This is similar to the logic of target networks in deep RL. Eq. 6 shows that optimizing EPE is equivalent to maximizing value up to a constant (the frozen value estimate); this means that an agent maximizing EPE will try to improve its expected future rewards. However, if the value estimate is perfect for a state, EPE is 0; this means that the agent will not try to reach high-value states that it knows are high value. EPE is subjective in the sense that the same value function can lead to different utilities depending on the agent’s value estimate. The essence of EPE is that agents are attracted to positive surprise. This still requires agents to figure out how to get to the goal, but once they arrive, the goal is “played out”—the goal itself is no longer of interest. For example, once you reach the end of a maze, it’s no longer of any interest to stick around and relive the victory, but that’s precisely what a value-maximizing agent would do if the end point is rewarding and the game doesn’t terminate. For the same reason, it’s of no interest to repeatedly retrace the path to victory, but that’s precisely what a value-maximizing agent would do if given the opportunity. A slightly more complicated model combines value and EPE, to accommodate the fact that even in the limit of perfectly learned values (where EPE is 0) agents may still prefer policies with higher values: \displaystyle\alpha V^{\pi}_{g}(s)+(1-\alpha)U^{\pi}_{g}(s)=V^{\pi}_{g}(s)-(1-\alpha)\hat{V}^{\pi}_{g}(s), (7) where \alpha\in[0,1] is a weighting parameter ( \alpha=0 recovers the EPE model; \alpha=1 recovers the value model). In the limit where \hat{V}^{\pi}_{g}=V^{\pi}_{g} , the model reduces to \alpha V^{\pi}_{g} (i.e., a dampened version of the value model). When \alpha is close to 0, optimal policies will pursue error maximization until learning has eliminated most sources of error, at which point policies will pursue value maximization. Principle 2: agents select goals that maximize expected prediction error Principle 1 concerned an agent’s “inner loop” where the goal is fixed and the policy is optimized. Principle 2 concerns an agent’s “outer loop” where the policy is fixed and the goal is optimized. We propose that the same objective function, EPE, is used in the outer loop: \displaystyle g^{\ast}=\operatornamewithlimits{argmax}_{g}U^{\ast}_{g}(s_{0}), (8) where s_{0} is the starting state and U^{\ast}_{g} is the EPE under the optimal policy \pi^{\ast}_{g} . 3