The Laplacian Keyboard: Beyond the Linear Span Siddarth Chandrasekar, Marlos C. Machado Abstract Across scientific disciplines, Laplacian eigenvectors serve as a fundamental basis for simplifying complex systems, from signal processing to quantum mechanics. In reinforcement learning (RL), these eigenvectors provide a natural basis for approximating reward functions; however, their use is typically limited to their linear span, which restricts expressivity in complex environments. We introduce the Laplacian Keyboard (LK), a hierarchical framework that goes beyond the linear span. LK constructs a task-agnostic library of options from these eigenvectors, forming a behavior basis guaranteed to contain the optimal policy for any reward within the linear span. A meta-policy learns to stitch these options dynamically, enabling efficient learning of policies outside the original linear constraints. We establish theoretical bounds on zero-shot approximation error and demonstrate empirically that LK surpasses zero-shot solutions while achieving improved sample efficiency compared to standard RL methods. 1 Introduction Primary causes are unknown to us; but are subject to simple and constant laws, which may be discovered by observation, the study of them being the object of natural philosophy. - Joseph Fourier This observation from Fourier 1878 highlights a recurring objective in science: finding simple laws that explain complex phenomena . Across disciplines, this often involves decomposing systems into fundamental components that reveal underlying structure. Fourier’s analysis of heat diffusion demonstrated that temperature evolution can be expressed as independent modes, each an eigenfunction of the Laplacian operator with its own decay rate. Similarly, the Fourier transform decomposes signals into sinusoidal components—also Laplacian eigenvectors—enabling efficient filtering and compression. These examples share a common thread: the eigenvectors of the Laplacian provide a natural basis that simplifies analysis and computation. In this work, we explore this principle in RL, showing that the eigenvectors of the Laplacian can serve as a principled basis for RL and how they can be leveraged to enable sample-efficient learning across tasks. Prior work has explored bases that can be linearly composed to efficiently solve new tasks, including bases over value functions ( Mahadevan 2005 ; Bellemare et al. 2019 ) , policies ( Agarwal et al. 2025b ) , options ( Barreto et al. 2019 ) and state visitation distributions ( Touati et al. 2023 ) . These methods derive their bases from two seemingly distinct perspectives: representations and behaviors . Figure 1 : Overview of the Laplacian Keyboard. ( Left ) A reward-free dataset is used to learn a representation basis and a USFA, inducing a behavior basis, which are directions in a latent space. For illustration, in this figure, we depict those as behaviours following straight-line trajectories . ( Right ) In a downstream navigation task, an agent (the boat) executes these options to interact with the environment. A zero-shot policy picks and executes one single option, producing a potentially suboptimal trajectory (red), whereas the LK trains a meta-policy that sequentially selects and switches between options, composing a piecewise trajectory (blue) that better matches the task. Representation-based methods learn features to approximate a task-relevant quantity. Proto-Value Functions ( Mahadevan 2005 , PVF;) , an early RL example of this principle, employs graph Laplacian eigenvectors as a basis for value function approximation. Given a new task, the agent learns the linear weights that combine these eigenvectors to approximate the optimal value function. More recently, Forward-Backward representations ( Touati et al. 2023 , FB;) learn a low-rank representation of the successor measure of a policy class, which is then used to approximate reward functions. For any new task, the agent infers the weights that best approximate the corresponding reward function for this basis. If the learned representation spans the reward function, FB produces the optimal policy with no further learning involved. This enables FB to produce zero-shot solutions. However, this zero-shot solution is constrained by the linear span of the learned features—the policy can be suboptimal when the task lies outside this span. The second category learns a basis of behaviors ( Sutton et al. 1999 , or options;) rather than features. Given a new task, these methods use generalized policy improvement ( Barreto et al. 2017 , GPI;) by evaluating each option in the basis and combining them using linear weights, guaranteeing performance at least as good as that of the best individual option, and often better. The Option Keyboard framework ( Barreto et al. 2019 ) extends this idea by learning state-dependent weights, enabling the agent to dynamically stitch together options across states. This allows the agent to solve a broader class of tasks than is possible with a single fixed combination of the behavior basis. However, a central limitation is that the choice of the option basis is not well defined and typically relies on manual specification of either the options or their associated reward functions. In this work, we bridge representation- and behavior-based approaches through the Laplacian Keyboard (LK), a framework that exploits their coupling for option discovery. We show that graph Laplacian eigenvectors serve dual roles: as a basis for reward approximation and as a principled, task-agnostic source of behaviors ( Machado et al. 2023 ) . LK combines these eigenvectors, learned from an offline, reward-free dataset, with Universal Successor Feature Approximators ( Borsa et al. 2019 , USFA;) to guarantee zero-shot optimal policies for tasks whose rewards lie within the span of the learned basis. Many downstream tasks, however, fall outside this regime. To address this limitation, LK introduces efficient adaptation via sequential composition (see Figure 1 ). Rather than relying on a single linear combination of basis elements, LK decomposes complex tasks into simpler subproblems. Each USFA-induced policy becomes a reusable option in a library of task-agnostic behaviors, and a meta-policy learns to select and stitch these options efficiently. This hierarchical structure enables LK to solve tasks whose reward functions cannot be represented within the basis span. By unifying zero-shot optimality within the basis with efficient composition beyond it, LK overcomes key limitations of both representation- and behavior-based methods. This work makes two key contributions. (1) We establish graph Laplacian eigenvectors as an effective reward basis through empirical evidence and theoretical analysis, including value-function approximation error bounds that characterize the expressivity of a finite Laplacian basis. (2) We introduce the Laplacian Keyboard, a hierarchical framework that stitches Laplacian-induced behaviors via a meta-policy, outperforming zero-shot baselines while achieving superior sample efficiency compared to standard RL agents. 2 Background We first formalize the problem addressed by LK and then review related approaches that tackle the same problem by learning a basis of representations or behaviors. We use calligraphic capitals for sets (e.g., \mathscr{S},\mathscr{A} ), capitals for random variables (e.g., S_{t},A_{t} ), bold capitals for matrices (e.g., \mathbf{L},\mathbf{P}_{\!\pi} ), non-bold lowercase for functions (e.g., \pi,r ), and bold lowercase for vectors (e.g., \mathbf{w},\mathbf{e} ). Functions over the state or action space are often represented in vectorized form by indexing their values, denoted using bold notation (e.g., \mathbf{r} ). 2.1 Problem Definition In RL, the agent-environment interaction is modeled as a Markov Decision Process (MDP) M=(\mathscr{S},\mathscr{A},p,r,\gamma) , with state space \mathscr{S} , action space \mathscr{A} , transition dynamics p(\cdot\mid s,a) , and discount factor \gamma . In this work, we define the reward r as a function of the next state. Specifically, for any transition (s,a,s^{\prime}) , the state-based reward is given by r(s^{\prime})=\int_{s,a}p(s^{\prime}\mid s,a)\,r(s,a,s^{\prime})\,ds\,da . The goal is to learn a policy \pi:\mathscr{S}\to\Delta(\mathscr{A}) that maximizes discounted expected return, expressed via the action-value function q_{\pi}(s,a)=\mathbb{E}_{\pi}\Big[\sum_{t=0}^{\infty}\gamma^{t}r(S_{t+1})\mid S_{0}=s,A_{0}=a\Big] , where S_{t+1}\sim p(\cdot\mid S_{t},A_{t}) . In this work, we aim to learn a basis of behaviors that an agent can employ to output near-optimal policies for any given reward function in a sample-efficient manner. Similar to Touati et al. 2023 and Agarwal et al. 2025b , we formulate this objective as a two-phase process, commonly referred to as Unsupervised RL . In the pre-training phase, the agent interacts with the environment for M steps to learn the basis without access to any reward function. In the subsequent downstream phase, tasks are specified through reward functions, and the agent must leverage its prior knowledge to solve these tasks within N steps, where N\ll M . 2.2 Solution Methods In this section, we provide a brief overview of related unsupervised RL methods, organized by whether they use representations or options as the underlying basis. A more detailed review is given in Appendix A . Representation-based methods learn a state representation \phi:\mathscr{S}\rightarrow\mathbb{R}^{d} of dimension d during pre-training, defining a representation basis to approximate downstream rewards as: r(s)\approx\mathbf{w}^{\top}\bm{\phi}(s),\quad\forall s\in\mathscr{S}. Simultaneously, a USFA is used to train a policy \pi(a\mid s,\mathbf{w}) ( Borsa et al. 2019 ) . In the downstream phase, once a task is specified, a weight vector \mathbf{w} is estimated from N samples via simple linear regression to approximate the reward function, which parameterizes the USFA. When the reward function lies exactly within the span of \bm{\phi} , the USFA recovers the optimal policy, making the choice of \bm{\phi} and the USFA training procedure critical (see Touati et al. 2023 ; Borsa et al. 2019 ; Ollivier 2025 for further discussion). This quick adaptation using only a few samples via linear regression is referred to as zero-shot learning by Touati et al. 2023 . Methods like Forward-Backward Representations ( Touati et al. 2023 ) and Proto Successor Measure ( Agarwal et al. 2025b ) model \bm{\phi} as a low-rank representation of the successor measure of a class of policies. These approaches have been shown to scale to large state and action spaces with minimal human prior ( Tirinzoni et al. 2025b ) . The primary limitation of these approaches is expressivity: the linear span of \bm{\phi} must be rich enough to approximate all potential downstream rewards, often requiring a high-dimensional d to cover a diverse set of complex tasks. Option-based methods take a complementary approach by learning a reusable option basis that can be linearly combined to solve new tasks. Most methods assume access to reward features \phi:\mathscr{S}\rightarrow\mathbb{R}^{d} , which define this basis: each option is trained to be optimal for a reward induced by these features. For downstream rewards of the form r(s)=\mathbf{w}^{\top}\bm{\phi}(s) , options can be combined via General Policy Evaluation and Improvement ( Barreto et al. 2020 , GPE & GPI;) , yielding a policy at least as good as any individual option. We refer to Borsa et al. 2019 for details on GPE & GPI and its connection to the USFA framework. The Option Keyboard (OK) framework ( Barreto et al. 2019 ) generalizes this approach by learning a meta-policy \pi_{\text{OK}} that composes options using state-dependent weights at execution time, rather than a single fixed combination, enabling solutions to a broader class of tasks ( Chandrasekar & Machado 2025 ) . However, OK has two key limitations: (1) it relies on manually specified reward features \bm{\phi} , and (2) lacks a principled procedure for constructing the option basis. While Alegre et al. 2025 partially addresses the latter, dependence on handcrafted features remains a central limitation when such features are difficult to specify. We discuss this work further in Section 7 . 3 The Laplacian Basis Figure 2 : We illustrate the Laplacian basis in the Four-Room environment, a toy domain introduced by Sutton et al. 1999 . The hotter (redder) the state’s color, the larger the corresponding entry; color scales are normalized independently for each subplot. (a) Selected eigenvectors of the graph Laplacian induced by a uniform random policy, with values in parentheses indicating the eigenvector graph norm (Eqn. 1 ). (b) A sample reward function and its reconstruction using the first k eigenvectors. Values in parentheses denote the mean squared error between the original and reconstructed reward functions. Increasing the basis size improves reconstruction accuracy. This section motivates the use of graph Laplacian eigenvectors as a basis for approximating reward functions. We restrict ourselves to MDPs in which the induced Markov chain under any policy of interest is reversible ( Levin & Peres 2017 ) . Any such MDP can be modeled as an undirected graph, with states as vertices and transitions as edges. 1 1 1 See Appendix B.1.1 for an in-depth background. In the discrete case, the graph Laplacian is defined as \mathbf{L}\,\dot{=}\,\mathbf{I}-\mathbf{P}_{\!\pi} , where \mathbf{P}_{\!\pi}(s,s^{\prime})=\sum_{a\in\mathscr{A}}\pi(s,a)p(s^{\prime}|s,a) denotes the probability of going from state s to s^{\prime} under policy \pi . Let \mathbf{e}_{i} denote the i -th eigenvector of \mathbf{L} , with eigenvector ordering defined by their corresponding eigenvalue. These eigenvectors encode the structural properties of the state space in a natural order: eigenvectors associated with smaller eigenvalues are smoother, ranging from the constant vector \mathbf{e}_{1} to progressively more oscillatory patterns as the eigenvalue increases (see Figure 2 a). To formalize this notion of smoothness, we rely on the graph norm ( Zhu & Rabbat 2012 ) ; for any function f:\mathscr{S}\to\mathbb{R} defined over the state space: \|f\|_{G}\,\dot{=}\,\left[\frac{1}{2}\sum_{(i,j)\in\mathscr{S}\times\mathscr{S}}\mathbf{P}_{\!\pi}(i,j)(f(i)-f(j))^{2}\right]^{\frac{1}{2}}. (1) The graph norm penalizes local variations weighted by the transition probabilities, smaller values indicating that the signals vary smoothly across connected states (see Figure 2 a ). The Laplacian representation is the mapping \phi:\mathscr{S}\rightarrow\mathbb{R}^{k} ( 00 is a hyper-parameter. The notation \llbracket\mathbf{u}_{k}\rrbracket denotes a stop-gradient operation. For notational simplicity, we henceforth use \mathbf{e}_{i} to denote the learned eigenvectors \mathbf{u}_{i} obtained from this optimization. We use the Laplacian representation \bm{\phi}(s)\in\mathbb{R}^{k} to parameterize a family of reward functions of the form r_{\mathbf{w}}(s)=\mathbf{w}^{\top}\bm{\phi}(s) . To model policies and value functions for this reward family, we train a Universal Successor Feature Approximator (USFA) ( Barreto et al. 2017 ) , which generalizes successor representations ( Dayan 1993 ) by conditioning on the reward parameter \mathbf{w} . The USFA consists of a policy \pi_{\mathbf{w}} and a corresponding successor feature estimator \bm{\psi}_{\mathbf{w}} . The successor features encode the expected discounted accumulation of Laplacian features under the conditioned policy: \bm{\psi}(s,a,\mathbf{w})=\mathbb{E}_{\pi_{\mathbf{w}}}\!\left[\sum_{t=0}^{\infty}\gamma^{t}\bm{\phi}(S_{t+1})\,\middle|\,S_{0}=s,\,A_{0}=a\right]. (3) Training is performed off-policy using transitions (s,a,s^{\prime}) sampled from a dataset \mathscr{D} with empirical dynamics p_{\mathscr{D}}(\cdot\mid s,a) . During training, weight vectors \mathbf{w} are sampled and used to jointly condition both the policy and the successor feature estimator. The successor features are learned by enforcing Bellman consistency, \bm{\psi}(s,a,\mathbf{w})=\bm{\phi}(s^{\prime})+\gamma\mathbb{E}_{a^{\prime}\sim\pi_{\mathbf{w}}(\cdot\mid s^{\prime})}\bm{\psi}(s^{\prime},a^{\prime},\mathbf{w}), (4) where s^{\prime}\sim p_{\mathscr{D}}(\cdot\mid s,a) . Under exact representation, the USFA recovers optimal policies for all reward functions lying in the span of the k Laplacian eigenvectors, with the optimal action-value function given by q^{*}_{\mathbf{w}}(s,a)=\mathbf{w}^{\top}\bm{\psi}(s,a,\mathbf{w}) . In continuous action spaces, \pi_{\mathbf{w}} is optimized using policy gradients with this value estimate, while in discrete action spaces actions are selected greedily via \arg\max_{a}\mathbf{w}^{\top}\bm{\psi}(s,a,\mathbf{w}) ( Borsa et al. 2019 ) . During training, following Touati et al. 2023 , weight vectors \mathbf{w} are sampled from a broad distribution to promote wide coverage of the reward-function space. For any reward function in the linear span of the eigenvectors, the USFA can output the optimal policy. For reward functions outside this span, we have provided a bound for the sub-optimality of the resulting policies in Theorem 3.1 . The next section describes how LK goes beyond the linear span to address a larger family of reward functions. 4.2 Downstream Phase In the downstream phase, a meta-policy, \pi_{\text{LK}}:\mathscr{S}\to\mathbb{R}^{k} , is trained to sequence options for solving arbitrary tasks. This meta-policy generates weight vectors \mathbf{w}\in\mathbb{R}^{k} that parameterize the USFA, which in turn produces and executes options. The hierarchical control mechanism operates as follows: \pi_{\text{LK}} outputs a weight vector \mathbf{w} that parameterizes the USFA to instantiate an option; this option executes for a predetermined horizon t_{\text{term}} or until its termination condition is satisfied; control then returns to \pi_{\text{LK}} , which selects a new weight vector based on the new state. This process continues iteratively until the task is solved. When the downstream reward function r lies within the span of the learned eigenvectors \mathbf{e} , the meta-policy recovers the optimal weight vector \mathbf{w} satisfying r(s)=\mathbf{w}^{\top}\bm{\phi}(s) , achieving zero-shot transfer as in other representation-based methods. For out-of-span rewards, the meta-policy approximates the optimal policy by sequencing multiple options ( Alegre et al. 2025 ) , stitching locally optimal behaviors induced by different weight vectors into a near-optimal global policy (see Figure 4 ). This compositional structure enables LK to address reward functions beyond the linear span of the learned features. Figure 4 : The LK stitches behaviors from the learned basis to approximate the optimal policy when the reward function lies outside the span of the eigenvectors. We use the same reward function r from Figure 2 . The first panel contains the optimal policy of the original reward function to reach the goal state. The second panel displays the optimal policy of the reconstructed reward function with k=6 eigenvectors ( {r}_{6} ). This policy fails to reach the goal state. The last three panels show how LK stitches three individual options, each terminated after a fixed horizon t_{\text{term}}=6 . Together, these options form a near-optimal policy that closely replicates the true optimal behavior through a hierarchical control loop. When compared to prior option-based methods, LK does not rely on manually specified reward features or predefined training tasks to form the behavior basis. Instead, it is learned directly from the offline dataset. Consequently, the basis scales with the coverage of the dataset and the chosen value of k . In addition, we do not constrain the basis to a finite set of options; the USFA formulation provides a continuous library of option policies. Finally, we expect LK to achieve better sample efficiency than flat agents, i.e., agents that act directly in the primitive action space, for two reasons. (1) By learning over options rather than primitive actions, LK propagates value information across multiple timesteps, which can accelerate credit assignment ( Sutton et al. 1999 ) . (2) Exploration in this structured space corresponds to exploring over coherent, temporally-extended behaviors ( Machado et al. 2017 ; Machado et al. 2023 ; Jinnai et al. 2019 ) , which can reduce variance and enable faster downstream learning. However, unlike previous approaches, here the action space of \pi_{\text{LK}} is the continuous space of weight vectors \mathbf{w} that parameterize options. This induces a smooth control space in which small changes in the policy output lead to similar behaviors, instead of having small parameter changes producing qualitatively different action sequences. 5 Laplacian Behavior Basis Here we visualize the behaviors induced by the graph Laplacian eigenvectors in the DeepMind Control suite ( Tassa et al. 2018 , DMC;) . We consider three domains: Cheetah , Quadruped , and Walker —corresponding to agents with distinct morphologies and control dynamics. Observations consist of joint positions and velocities; additional environment details are provided in Appendix C.1 . We use the offline dataset from Yarats et al. 2022 , which contains trajectories collected under diverse exploration strategies. Figure 5 illustrates the behaviors induced by the first Laplacian eigenvector across the three domains, corresponding to the negative ( \mathbf{w}_{-\mathbf{e}{{}_{1}}} ) and positive ( \mathbf{w}_{+\mathbf{e}{{}_{1}}} ) directions of that eigenvector. Although the specific behaviors differ across environments, the two directions consistently induce opposing outcomes. In Cheetah , the negative direction produces forward hopping, while the positive direction results in an inverted posture. In Quadruped , the negative direction yields an inverted stance, whereas the positive direction induces an upright posture. In Walker , the negative direction leads to a backward headstand, while the positive direction produces a forward headstand. We observe similar oppositional structure for other eigenvector directions and across multiple runs. These behaviors emerge without hand-crafted priors, providing a scalable mechanism for learning a behavioral basis without explicitly enumerating skills. Figure 5 : Instances from the Laplacian behavior basis. The left panel shows the behavior corresponding to \mathbf{w}_{-\mathbf{e}{{}_{1}}}=[-1,0,0,\dots] , while the right panel shows the behavior corresponding to \mathbf{w}_{+\mathbf{e}{{}_{1}}}=[+1,0,0,\dots] . 6 Empirical Validation on DMC Building on the preceding analysis, we empirically examine two key properties of the Laplacian behavior basis and its use within LK. (1) Laplacian eigenvectors provide a task-agnostic representational basis that supports zero-shot RL. (2) The hierarchical structure of LK enables sample-efficient learning that extends beyond what can be achieved through linear combinations of the basis alone. Environment: We consider four tasks in each of the DMC domains: Cheetah ( Run, Run Backwards, Walk, Walk Backwards ), Quadruped ( Jump, Run, Stand, Walk ), and Walker ( Flip, Run, Stand, Walk ). Backward variants are denoted with a -B suffix (e.g., Run-B ). Each domain includes offline datasets from three exploration policies: APS , Proto , and RND ( Yarats et al. 2022 ) . The diversity of domains, tasks, and data-collection strategies makes DMC a suitable benchmark for evaluating the representational and learning properties of LK. Implementation: For each domain–dataset pair and a given basis size k , we train a Laplacian encoder and a corresponding USFA policy, resulting in 9 pre-trained agents that are used for zero-shot evaluation. Using these pre-trained components, we then train a meta-policy for each downstream task, yielding a total of 36 meta-policies (9 domain–dataset pairs \times 4 tasks). We repeat this procedure for basis sizes k\in\{1,2,3,5,10,20,50\} . Both the USFA and the meta-policies are trained using TD3 ( Fujimoto et al. 2018 ) . Zero-shot performance is evaluated using 10K labeled samples, while each meta-policy is trained for 200K environment interactions. To contextualize task difficulty, we additionally train a flat TD3 agent for 2M interactions. Both the meta-policy and the flat baseline are evaluated every 10K interactions over 10 episodes, and all results are averaged over 30 independent runs. Table 1 : Comparison of zero-shot performance between FB, LK ( k=50 ), and a flat TD3. Reported values are the mean \pm standard error, averaged over 30 independent runs for each of the three datasets (90 runs total per entry). Task FB LK Flat TD3 Cheetah (avg) 552 450 786 Run 248 {\scriptstyle\pm 7} 196 {\scriptstyle\pm 9} 714 {\scriptstyle\pm 13} Run-B 229 {\scriptstyle\pm 4} 188 {\scriptstyle\pm 6} 471 {\scriptstyle\pm 15} Walk 819 {\scriptstyle\pm 15} 709 {\scriptstyle\pm 22} 980 {\scriptstyle\pm 2} Walk-B 912 {\scriptstyle\pm 10} 706 {\scriptstyle\pm 14} 981 {\scriptstyle\pm 1} Quadruped (avg) 470 509 592 Jump 483 {\scriptstyle\pm 7} 554 {\scriptstyle\pm 10} 625 {\scriptstyle\pm 38} Run 317 {\scriptstyle\pm 6} 366 {\scriptstyle\pm 5} 456 {\scriptstyle\pm 31} Stand 617 {\scriptstyle\pm 12} 705 {\scriptstyle\pm 14} 708 {\scriptstyle\pm 35} Walk 461 {\scriptstyle\pm 9} 410 {\scriptstyle\pm 11} 578 {\scriptstyle\pm 48} Walker (avg) 584 582 780 Flip 412 {\scriptstyle\pm 13} 507 {\scriptstyle\pm 15} 667 {\scriptstyle\pm 7} Run 356 {\scriptstyle\pm 7} 294 {\scriptstyle\pm 9} 571 {\scriptstyle\pm 15} Stand 754 {\scriptstyle\pm 13} 635 {\scriptstyle\pm 23} 977 {\scriptstyle\pm 2} Walk 816 {\scriptstyle\pm 8} 890 {\scriptstyle\pm 11} 905 {\scriptstyle\pm 17} 6.1 Generality of the Laplacian Basis We first assess whether the Laplacian basis can linearly approximate arbitrary reward functions through zero-shot evaluation. For comparison, we use Forward-Backward representations (FB), which has shown great promise ( Touati et al. 2023 ; Tirinzoni et al. 2025b ) with theoretical guarantees ( Blier et al. 2021 ) . Accordingly, we expect LK’s zero-shot performance to be atleast comparable with FB and competitive with a strong oracle up to a constant factor. Table 1 presents zero-shot performance for a basis size of k=50 , with results averaged across the three datasets unless noted otherwise. We focus on k=50 here, with additional results provided in Appendix D.2 , as this basis size was used in Touati et al. 2023 , where performance was reported to plateau for larger values of k . We use a task-specific flat TD3 agent as an oracle baseline. LK achieves strong zero-shot performance comparable to FB; a sign test finds no significant difference between the two methods, indicating that the Laplacian basis matches the expressiveness of this established alternative. Relative to the oracle, LK attains at least 75% of oracle return on 8 of 12 tasks, indicating effective approximation of a broad class of reward functions. This behavior is particularly pronounced in the Quadruped domain, where LK achieves near-oracle performance across all tasks. For clarity, we report only LK’s zero-shot results in subsequent sections to facilitate direct comparison with downstream performance. Figure 6 further presents LK’s zero-shot performance as a function of the basis size. In Cheetah and Walker , performance generally improves with increasing basis size. For most tasks, improvements are gradual, while a small number exhibit sharp gains, suggesting that those additional basis components typically contribute incrementally, with occasional dominant directions. In contrast, Quadruped achieves its strongest performance with the smallest basis ( k=1 ), with performance degrading slightly as the basis size increases. This suggests that the leading eigenvector captures most of the task-relevant structure, while higher-order components provide limited additional benefit. We observe that the first eigenvector induces behaviors that regulate torso elevation, central to the reward functions of all four tasks, and discuss this effect in detail in Appendix E . Figure 6 : Zero-shot performance of LK for varying basis sizes, averaged over 30 independent runs for each of the three datasets (90 runs total per curve). 6.2 Hierarchical Composition We next examine the benefits of LK’s hierarchical structure. Specifically, we show that hierarchy (1) enables LK to improve upon zero-shot solutions with minimal additional interaction and (2) improves sample efficiency relative to flat agents. These experiments are informative not because the basis is rich, but because it is deliberately constrained ( k\in\{1,2,3,5,10\} ), serving as a proxy for settings in which even larger bases may be insufficient. Table 2 reports the percentage improvement of LK after 200K environment interactions relative to its zero-shot performance. In Cheetah and Walker , LK improves over the zero-shot solution on most tasks and across most basis sizes. The improvements are largest when the behavioral basis is small, highlighting the regime in which hierarchical composition is most effective. When k=1 , the USFA can represent only two options, corresponding to the positive and negative directions of the eigenvector, since no linear combinations are possible and scaling the weight vector has no effect. Despite this severe restriction, the meta-policy’s ability to switch between these two options yields improvements exceeding 100% on several tasks. This illustrates a key property of LK: even highly constrained bases can be effective when coupled with hierarchical control. As k increases, the magnitude of improvement generally decreases but remains largely positive, indicating that the benefits of composition are strongest when tasks require non-linear reuse of a limited set of basis behaviors. Table 2 : Relative percentage improvement of LK over its zero-shot baseline for varying basis sizes, estimated from 30 independent runs for each of the three datasets (90 runs total per entry). Improvement (%) for Basis Size k Task 1 2 3 5 10 20 50 cheetah (avg) 1296 2553 288 97 51 17 10 Run 1081 4608 176 61 58 44 34 Run-B 1678 1241 508 154 64 -3 1 Walk 808 3532 125 41 29 21 12 Walk-B 1619 829 345 133 53 5 -6 quadruped (avg) -3 -12 -11 -17 -14 -2 -12 Jump -3 -13 -13 -19 -26 -7 -13 Run -3 -11 -13 -21 -15 -2 -17 Stand 1 -16 -14 -20 -16 -7 -13 Walk -6 -7 -5 -10 -1 7 -4 walker (avg) 112 107 115 132 159 62 35 Flip 106 99 104 115 189 53 46 Run 55 52 34 80 185 62 44 Stand 118 145 119 79 136 97 49 Walk 170 132 204 253 126 37 2 In contrast, Quadruped exhibits the opposite trend. LK rarely improves over its zero-shot solution across tasks and basis sizes. Consistent with observations from the previous section, tasks in this domain appear to be well approximated by the leading eigenvector. If the zero-shot weight vector is already near-optimal, hierarchical learning reduces to identifying a single weight vector in \mathbb{R}^{k} , which we hypothesize to be a challenging optimization problem. Nevertheless, LK can be constructed to choose between the zero-shot solution and hierarchical composition, guaranteeing performance no worse than the best of the two. Figure 7 : Evaluation curves for LK with varying basis sizes and a flat TD3 agent on the Walk task, averaged over 30 independent runs for each of the three datasets (90 runs total per curve). Figure 7 presents evaluation curves for LK and the flat TD3 on a representative task, Walk , across all three domains; the remaining curves are provided in the Appendix D.3 . We analyze the hierarchical advantages of LK in more detail in the Walker domain with k=20 . The yellow shaded region indicates the sample-efficiency advantage of LK relative to the flat baseline. Both agents perform random exploration for the first 10K steps, after which learning begins. Despite operating in a higher-dimensional action space (dimension k ) than the flat agent (domain-specific action dimension, 6 in Walker ), LK exhibits rapid performance gains and converges to a stable policy earlier. This behavior is consistent with the hypothesis outlined in Section 4.2 . We additionally indicate the hierarchical composition gain, defined as the improvement of LK over its zero-shot performance, corresponding to the values reported in Table 2 . 7 Comparison Against Privileged Baseline As a final empirical evaluation, we compare LK to Option Keyboard Basis ( Alegre et al. 2025 , OKB;) . OKB constructs its option basis from environment-provided reward features that span downstream tasks and is theoretically guaranteed to recover a basis sufficient for optimal control. This comparison assesses whether LK can achieve competitive performance without access to such privileged, handcrafted reward information, thereby highlighting the effectiveness of the Laplacian behavior basis. We evaluate on the Item-Collector task ( Barreto et al. 2019 ; Alegre et al. 2025 ) , a 10\times 10 toroidal gridworld where the agent must collect items in a fixed order. Transitions are annotated with reward features for OKB, while LK is pre-trained on a reward-free dataset collected via a uniform random walk. The USFA is trained with Double DQN ( van Hasselt et al. 2016 ) using basis sizes k\in\{20,30,40\} . Zero-shot performance is evaluated with 50K labeled samples, and the meta-policy is trained for 500K steps. Figure 8 compares the evaluation returns of OKB and LK. Despite relying only on transition data and lacking privileged reward features, LK achieves performance close to OKB. With more training steps, we observe that the gap between OKB and LK decreases, indicating that LK can recover structural information from environment dynamics alone to induce effective behaviors. More generally, it suggests that strong downstream performance does not require explicit reward-feature engineering, as Laplacian eigenvectors form a scalable basis for reusable options. 8 Conclusion We establish both theoretically and empirically that graph Laplacian eigenvectors form a principled and sample-efficient basis for RL. When combined with a USFA, they induce a task-agnostic behavior basis that can be queried to solve a range of downstream tasks, effectively functioning as a behavioral foundation model. However, as highlighted by the big world hypothesis ( Javed & Sutton 2024 ) , no fixed basis can capture the full diversity of tasks in large and complex environments. To address this limitation, we introduce the Laplacian Keyboard, a hierarchical framework in which a meta-policy adapts the fixed behavior basis to a given task through sequential composition at execution time. As noted by Javed & Sutton 2024 , “the key research challenge for achieving goals in big worlds is to come up with solution methods that efficiently utilize the limited resources of the agent” ; LK follows this principle by treating the learned basis as a structured starting point rather than a complete solution. Empirical results across multiple environments and basis sizes demonstrate improved performance over zero-shot baselines and greater sample efficiency than flat RL agents. Figure 8 : Evaluation curves for LK with varying basis sizes and OKB, averaged over 30 independent runs. The shaded region indicates the standard error. The horizontal dashed line corresponds to the zero-shot performance. While this work focuses on an offline, reward-free setting, a natural extension is a fully online framework in which representation-driven option discovery ( Machado et al. 2023 ; Klissarov & Machado 2023 ) is combined with LK to support efficient exploration and continual refinement of the behavior basis. We also observe that tasks favor different option termination horizons t_{\text{term}} , suggesting that learning termination sets could further improve temporal abstraction and meta-policy adaptation. Impact Statement This paper presents work whose goal is to advance the field of Reinforcement Learning. There are many potential societal consequences of our work, none which we feel must be specifically highlighted here. Appendix A Extended Related Work The Laplacian Keyboard bridges representation-based and behavior-based solution methods to address the zero-shot RL problem. In this section, we review closely related work along both directions, extending the discussion beyond that provided in the main paper. Representation-based Methods: A common approach to zero-shot RL is to learn task-agnostic state representations that can be reused for downstream control. Examples include Hilbert Foundation Policies ( Park et al. 2024 , HILP;) , Proto Successor Measures ( Agarwal et al. 2025b , PSM;) , Regularized Latent Dynamics Prediction ( Jajoo et al. 2025 , RLDP;) , and TD-JEPA ( Bagatella et al. 2025 ) . While these methods differ in their objectives, they all aim to construct representations that support policy transfer without additional environment interaction. HILP learns representations sufficient for expressing goal-reaching value functions and uses them to construct intrinsic reward functions for training a latent-conditioned policy. PSM learns a basis over state visitation distributions, enabling downstream behaviors to be synthesized as linear combinations of these bases and supporting zero-shot policy construction via reward-dependent recombination. RLDP and TD-JEPA learn predictive latent representations by modeling future states while regularizing against feature collapse; these representations are then paired with USFA to enable zero-shot transfer. A unified treatment of several representation-based methods is provided by Agarwal et al. 2025a . Closely related in spirit to LK are the LoLa and ReLa algorithms proposed by Sikchi et al. 2025 , which also exploit pre-trained knowledge to improve downstream task performance. However, unlike the LK, these methods are not hierarchical in nature and are primarily motivated by reducing approximation errors arising during pre-training and inference. As a result, they are better characterized as fine-tuning or correction mechanisms rather than frameworks for skill composition. Behavior-based Methods: Set of Independent Policies ( Alver & Precup 2022 , SIP;) , Set Max Policies ( Zahavy et al. 2021 , SMP;) , and the Successor Feature Keyboard ( Carvalho et al. 2023 , SFK;) are few examples that focus on learning a reusable set of behaviors or policies to enable zero-shot control. SIP assumes access to a set of independent reward features and learns a policy for each feature. At test time, GPE & GPI are used to recover the optimal policy for any reward expressed as a linear combination of these features. SMP retains the linear reward assumption but optimizes for robustness by learning policies that maximize worst-case performance over admissible rewards; zero-shot transfer is again achieved using GPE & GPI. SFK relaxes the assumption of known reward features and learns a policy basis by extracting shared structure across a set of training tasks. A unifying backbone of these methods is the successor representation ( Dayan 1993 , SR;) and its generalization, successor features ( Barreto et al. 2017 , SF;) . These frameworks decompose the value function into a dynamics-dependent component, capturing long-term state visitation, and a task-specific component defined by the reward. SR represents each state by its expected discounted future state occupancies under a policy, while SF extends this idea to feature spaces by modeling expected discounted feature activations. Under the assumption of linear reward functions, this decomposition allows value functions to be computed as inner products between SF and reward weights, naturally enabling zero-shot transfer via GPE & GPI and USFA. Appendix B Laplacian Basis In this section, we provide the proof for Theorem 3.1 . We first provide the necessary background, followed by the proof. We reintroduce a few details for completeness. B.1 Background The proof and theorem is partly inspired from the spectral graph theory literature. Hence we first introduce concepts in the language of graph theory before using them for RL. B.1.1 Spectral Theory Graph Laplacian: Consider an undirected graph G=(\mathscr{V},\mathscr{E}) with vertex set \mathscr{V} and edge set \mathscr{E} . Each edge (i,j)\in\mathscr{E} carries a non-negative weight w_{ij}\geq 0 quantifying the connection strength between vertices i and j . These weights define the weighted adjacency matrix \mathbf{W}\in\mathbb{R}^{|\mathscr{V}|\times|\mathscr{V}|} , where \mathbf{W}_{ij}=w_{ij} for (i,j)\in\mathscr{E} and \mathbf{W}_{ij}=0 otherwise. The vertex degrees d_{i}=\sum_{j}w_{ij} form the degree matrix \mathbf{D}=\mathrm{diag}(d_{1},d_{2},\dots,d_{|\mathscr{V}|}) . For undirected graphs, the adjacency matrix exhibits symmetry: w_{ij}=w_{ji}\,\forall(i,j)\in\mathscr{E} . The graph Laplacian matrix is defined as \mathbf{L}\,\dot{=}\,\mathbf{D}-\mathbf{W} , a real symmetric positive semi-definite matrix admitting the eigendecomposition \mathbf{L}=\mathbf{E}\mathbf{\Lambda}\mathbf{E}^{\top} . Here, \mathbf{E}=[\mathbf{e}_{1},\mathbf{e}_{2},\dots,\mathbf{e}_{|\mathscr{V}|}] contains orthonormal eigenvectors and \mathbf{\Lambda}=\mathrm{diag}(\lambda_{1},\lambda_{2},\dots,\lambda_{|\mathscr{V}|}) contains the corresponding non-negative eigenvalues ordered such that 0=\lambda_{1}\leq\lambda_{2}\leq\cdots\leq\lambda_{|\mathscr{V}|} . These eigenvectors encode structural properties of the graph in an ordered fashion: eigenvectors associated with smaller eigenvalues (low frequencies) exhibit smooth variations across the graph, while those associated with larger eigenvalues (high frequencies) capture rapid oscillations between neighboring vertices. Graph Fourier Transform: Any graph signal f:\mathscr{V}\rightarrow\mathbb{R} defined over the vertices admits an exact decomposition as a linear combination of the Laplacian eigenvectors: \mathbf{f}=\sum_{i=1}^{|\mathscr{V}|}\hat{f}_{i}\mathbf{e}_{i}, (5) where the Graph Fourier coefficients \hat{f}_{i}=\langle\mathbf{f},\mathbf{e}_{i}\rangle represent the projection of \mathbf{f} onto the i -th eigenvector. The Graph Fourier Transform (GFT) \mathbf{\hat{f}}=\mathbf{E}^{\top}\mathbf{f} maps signals from the vertex domain to the spectral domain, where \mathbf{\hat{f}}=[\hat{f}_{1},\hat{f}_{2},\dots,\hat{f}_{|\mathscr{V}|}]^{\top} denotes the signal’s spectral representation. This transformation serves as the graph analog of the classical Fourier transform, decomposing arbitrary graph signals into fundamental frequency components. Signals dominated by small coefficients \hat{f}_{i} for low i vary smoothly across the graph topology, while those with significant high-index coefficients exhibit complex, irregular patterns that vary sharply between connected vertices. A fundamental property of the GFT is energy preservation across domains, formalized by Parseval’s theorem ( Shuman et al. 2016 ) : Lemma B.1 (Parseval’s Theorem) . The energy of a signal in the vertex domain equals its energy in the spectral domain: \sum_{i=1}^{|\mathscr{V}|}f(i)^{2}=\sum_{i=1}^{|\mathscr{V}|}\hat{f}_{i}^{2}. (6) This property allows us to measure energy of the signal as a function of the Graph Fourier coefficients. Truncated Spectral Approximation: For large graphs where |\mathscr{V}| is large, computing and storing the full basis becomes computationally prohibitive. This constraint requires restricting the representation to a truncated spectral basis consisting of the first k\ll|\mathscr{V}| eigenvectors corresponding to the smallest eigenvalues. The resulting approximation \mathbf{f}_{k}=\sum_{i=1}^{k}\hat{f}_{i}\mathbf{e}_{i} (7) captures the signal’s low-frequency components—variations that are smooth with respect to the graph structure. This dimensionality reduction induces an inductive bias toward smoother signals that vary gradually across connected vertices while filtering out high-frequency fluctuations. The truncation sacrifices exact reconstruction in favor of computational efficiency. The approximation error induced by truncation can be quantified through Parseval’s theorem. Since the omitted eigenvectors correspond to the high-frequency components, the squared reconstruction error is given by \operatorname{error}(f,k)\dot{=}\sum_{i=k+1}^{|\mathscr{V}|}\hat{f}_{i}^{2}, (8) representing the energy concentrated in the discarded frequencies. This quantity measures the information loss incurred by restricting the reconstruction space to the subspace spanned by the first k eigenvectors. As k increases, the approximation error decreases, tightening the reconstruction quality. Graph Signal Smoothness: To formalize the notion of smoothness for signals defined over graphs, we rely on the graph total variation , which quantifies local variations weighted by edge connectivity ( Zhu & Rabbat 2012 ) . Definition B.2 (Graph Total Variation) . For any graph signal f:\mathscr{V}\to\mathbb{R} , the graph total variation is defined as \|f\|_{G}\,\dot{=}\,\left[\frac{1}{2}\sum_{(i,j)\in\mathscr{E}}w_{ij}(f(i)-f(j))^{2}\right]^{\frac{1}{2}}. (9) The graph total variation penalizes differences between signal values at connected vertices, weighted by their edge strengths. Smaller values of \|f\|_{G} indicate that the signal varies smoothly across connected vertices. The factor \frac{1}{2} accounts for double-counting in undirected graphs; Zhu & Rabbat 2012 omit this term by enumerating each edge exactly once. A signal f exhibits bounded variation if there exists a constant 0 Jump > Walk \approx Run . Tasks that introduce additional height- or velocity-dependent modulation require structure beyond the dominant basis direction, reducing the effectiveness of a minimal representation. Figure 17 : Absolute zero-shot weight magnitudes for all tasks, grouped by basis size. Each subplot shows all task-specific weights with low opacity, together with the mean absolute weight plotted as a thick line. Note that the entries in the x -axis are different indices for the weight magnitudes, there is no notion of time progression or size within a single panel; we connect the datapoints with a line to ease visualization. The first basis coefficient is indicated by a cross. Appendix F Pseudocode for training Laplacian Keyboard We present Python-style pseudocode for training the Laplacian encoder and USFA during pre-training, and the meta-policy during downstream learning. Low-level implementation details are omitted for clarity. ⬇ 1 def train_laplacian_encoder ( self ): 2 3 # sample contrastive state pairs 4 obs , pos_obs , neg_obs = self . sample_laplacian_batch () 5 6 # encode states 7 phi = self . encoder ( obs ) 8 phi_pos = self . encoder ( pos_obs ) 9 phi_neg = self . encoder ( neg_obs ) 10 11 # graph-smoothness loss 12 graph_loss = (( phi - phi_pos ) ** 2). mean () 13 14 # orthogonality-enforcing loss 15 error = compute_orthogonality_error ( phi , phi_neg ) 16 17 dual_loss = ( self . dual_vars . detach () * error ). sum () 18 barrier_loss = ( self . barrier_coefs . detach () * error . pow (2)). sum () 19 20 # optimize encoder 21 loss = graph_loss + dual_loss + barrier_loss 22 self . encoder_optimizer . step () ⬇ 1 def train_usfa ( self ): 2 3 # sample transition batch 4 obs , action , next_obs , done = self . sample_usfa_batch () 5 w = self . sample_task_weight () 6 7 # compute Laplacian features 8 phi = self . encoder ( obs ) 9 phi_next = self . encoder ( next_obs ) 10 11 # sample next action 12 next_action = self . actor ( phi_next , w ). sample () 13 14 # compute target successor features 15 psi_target = phi + (1 - done ) * self . gamma * \ 16 self . target_psi ( phi_next , next_action , w ) 17 18 # predict successor features 19 psi_pred = self . psi ( phi , action , w ) 20 21 # TD loss 22 psi_loss = mse ( psi_pred - psi_target . detach ()) 23 self . psi_optimizer . step () 24 25 # optionally update actor 26 if update_actor : 27 actor_loss = -( psi_pred * w ). sum (-1). mean () 28 self . actor_optimizer . step () 29 30 # update target networks 31 self . update_target_networks () ⬇ 1 def train_downstream ( self ): 2 obs = self . env . reset () 3 done = False 4 option_return = 0 5 option_length = 0 6 7 # sample initial option 8 w = self . meta_actor ( obs ) 9 10 while not done : 11 12 # act using USFA-conditioned policy 13 action = self . actor ( obs , w ). sample () 14 next_obs , reward , done = self . env . step ( action ) 15 16 # accumulate option return 17 option_return += ( self . gamma ** option_length ) * reward 18 option_length += 1 19 20 # terminate option 21 if option_length >= self . max_option_length or done : 22 self . replay_buffer . store ( obs , w , next_obs , 23 option_return , done ) 24 w = self . meta_actor ( next_obs ) 25 option_return = 0 26 option_length = 0 27 28 obs = next_obs 29 30 # update meta-policy 31 if update_meta : 32 batch = self . replay_buffer . sample () 33 self . update_critic ( batch ) 34 self . update_actor ( batch ) 35 self . update_target_networks ()