Inferring Implicit Goals Across Differing Task Models
Silvia Tulli, Stylianos Loukas Vasileiou, Mohamed Chetouani, Sarath Sreedharan · 2025
Read Inferring Implicit Goals Across Differing Task Models by Silvia Tulli, Stylianos Loukas Vasileiou, Mohamed Chetouani, Sarath Sreedharan (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2501-17704.txt.
Introduction
People rarely state everything they want. Silvia Tulli and co-authors study agents that must infer the unstated parts, especially when the user's mental model of the task differs from the agent's. In their setup, a user may wrongly assume some behavior is guaranteed, hiding an implicit subgoal. The method hunts bottleneck states, the chokepoints every good plan passes, as candidates for those hidden goals.
A querying strategy then asks the fewest possible questions to pin the goal down and certify a policy that reaches it. Experiments across tasks show the inference works. Value alignment, in this framing, is partly a conversation about what was never said.
Catalog record
| CREATOR | Silvia Tulli, Stylianos Loukas Vasileiou, Mohamed Chetouani, Sarath Sreedharan |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 46 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2501-17704
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Inferring Implicit Goals Across Differing Task Models Silvia Tulli, Stylianos Loukas Vasileiou, Mohamed Chetouani, Sarath Sreedharan Abstract One of the significant challenges to generating value-aligned behavior is to not only account for the specified user objectives but also any implicit or unspecified user requirements. The existence of such implicit requirements could be particularly common in settings where the user’s understanding of the task model may differ from the agent’s estimate of the model. Under this scenario, the user may incorrectly expect some agent behavior to be inevitable or guaranteed. This paper addresses such expectation mismatch in the presence of differing models by capturing the possibility of unspecified user subgoal in the context of a task captured as a Markov Decision Process (MDP) and querying for it as required. Our method identifies bottleneck states and uses them as candidates for potential implicit subgoals. We then introduce a querying strategy that will generate the minimal number of queries required to identify a policy guaranteed to achieve the underlying goal. Our empirical evaluations demonstrate the effectiveness of our approach in inferring and achieving unstated goals across various tasks. 1 Introduction Humans often omit details they consider obvious, unavoidable, or not worth mentioning when providing instructions. This omission leads to implicit goals and unstated preferences that AI systems must navigate to truly align with human intentions. One potential source of such unstated subgoals or preferences could be behaviors that the human user setting the agent objectives may identify as inevitable. The user would never bother stating anything regarding such behaviors, since they believe that it cannot be avoided. One class of such behavior is that of visiting bottleneck states in the context of goal-based Markov Decision Process (MDP). Here, bottleneck states refer to environment states that the agent must pass through to reach the stated goal. In many cases, the human user may want the agent to pass through or visit some bottleneck states in addition to the goal, thus forming a set of intermediate subgoals. However, the user may never specify them since, as far as the user is concerned, every trace of non-zero probability that leads to the goal passes through all the bottleneck states. This should be all well and good, provided the human bottleneck states are also bottleneck states for the agent. Otherwise, the agent must make an effort to figure out what the user’s underlying subgoals may be. To see how such problems may arise, consider an agent tasked with guiding a tourist to a famous art museum. The tourist simply says, “Get me a plan to get to the art museum,” unaware of the city’s metro system and expecting an above-ground route passing certain landmarks. The agent, however, might plan a route using the metro system. For the agent’s metro route, bottlenecks migh include entering the metro, making transfers, and exiting at the correct station. For the tourist’s expected route, they might include crossing a river and passing through the city center. This misalignment stems from differing world models: the agent’s comprehensive transit data versus the tourist’s limited knowledge of the city’s layout. The challenge in AI alignment lies in bridging this gap – identifying and accounting for implicit aspects of the task that weren’t mentioned. This paper explores how an agent can learn and achieve the implicit subgoals of another agent, particularly when their understandings of the environment differ. Traditional approaches to goal-conditioned reinforcement learning and planning often rely on explicit goal specification and assume shared world models between agents Liu et al. (2022) ; Chane-Sane et al. (2021) . This means trying to purely optimize for the achievement of specified human goals could lead to suboptimal or even incorrect task execution in complex, real-world scenarios. We propose a novel approach that introduces and formalizes a specific type of implicit subgoals within the MDP framework. Our method aims to compute policies that align with implicit subgoals by utilizing incomplete knowledge about another agent’s model of the world. We formalize the problem using two distinct MDPs: the executing agent’s model, the robot, and the goal-setting agent’s model, the human. We will use the bottlenecks in the human model as a basis for potential hypotheses about the implicit subgoals of the human. When potential implicit subgoals cannot be achieved, we will also make use of minimal querying to refine the agent’s hypotheses about the human subgoals. To evaluate our approach, we propose empirical evaluations in simulated scenarios. These evaluations will assess the ability to infer and align to implicit subgoals across various tasks. Our key contributions include: • Formulation of implicit subgoals within an MDP framework • Analysis of goal misalignment in the context of MDPs • Introduction of methods to reason about and infer unstated goals across differing world models • Development of novel querying algorithms for efficient information gathering to refine goal understanding • Creation of a framework to learn and satisfy implicit subgoals using MDPs and information-theoretic querying • Proposal of empirical evaluations in simulated scenarios to assess the ability to infer and align implicit subgoals across various tasks Figure 1: Bottleneck states are critical waypoints essential for reaching the goal in a given world model. Given a set of humans’ world models \mathbb{M}^{H} , the robot has to compute a policy \pi accounting for humans’ \mathbb{B} , as they might be candidates for human implicit subgoals. Whenever the robot cannot reach a human’s bottleneck due to discrepancies in world models, it queries whether this bottleneck is in fact a human subgoal. 2 Background The problems studied in the paper will be ones that can be modeled as infinite horizon discounted Markov Decision Processes (MDP), particularly ones where the primary objectives correspond to achieving some goals. Here, goal achievement corresponds to visiting specific goal states. The algorithms used in the paper will also exploit more traditional MDPs with rewards; as such, it is worth providing a more general definition. Specifically, an infinite horizon MDP is a tuple \mathcal{M}=\langle S,A,T,s_{0},\gamma,R\rangle , where S is the state space, A is the action space, T:S\times A\times S\rightarrow[0,1] is the transition function (e.g., T(s_{i},a_{i},s^{\prime}) provides the probability of transitioning from state s to s^{\prime} under action a ), R:S\to\mathbb{R} is the reward function, s_{0}\in S is the initial state of the agent, and \gamma\in[0,1) is the discount factor. Note that we will generally limit our attention to models where both S and A are finite sets. In this setting, the solution takes the form of a deterministic, stationary policy \pi:S\rightarrow A mapping states to actions. A value of a policy \pi , denoted as V^{\pi}:S\to\Re provides the expected cumulative discounted reward obtained by following the policy from a given state. A policy is considered optimal if no policy with a value higher than the current one exists. In this paper, we focus on goal-directed problems, where the set of goal states is given as S_{G}\subseteq S . In such problems, the reward function is sparse, i.e., it returns a small positive value for all states in S_{G} , and 0 otherwise. The states in S_{G} are treated as an absorbing state in that all transitions out of that state have zero probability. In such cases, we will represent the MDP