Unraveling the Hidden Dynamical Structure in Recurrent Neural Policies Jin Li, Yue Wu, Mengsha Huang, Yuhao Sun, Hao He, Xianyuan Zhan Abstract Recurrent neural policies are widely used in partially observable control and meta-RL tasks. Their abilities to maintain internal memory and adapt quickly to unseen scenarios have offered them unparalleled performance when compared to non-recurrent counterparts. However, until today, the underlying mechanisms for their superior generalization and robustness performance remain poorly understood. In this study, by analyzing the hidden state domain of recurrent policies learned over a diverse set of training methods, model architectures, and tasks, we find that stable cyclic structures consistently emerge during interaction with the environment. Such cyclic structures share a remarkable similarity with limit cycles in dynamical system analysis, if we consider the policy and the environment as a joint hybrid dynamical system. Moreover, we uncover that the geometry of such limit cycles also has a structured correspondence with the policies’ behaviors. These findings offer new perspectives to explain many nice properties of recurrent policies: the emergence of limit cycles stabilizes both the policies’ internal memory and the task-relevant environmental states, while suppressing nuisance variability arising from environmental uncertainty; the geometry of limit cycles also encodes relational structures of behaviors, facilitating easier skill adaptation when facing non-stationary environments. † † affiliationnotice: Equal contribution 1 Introduction Acting under partial observability and non-stationary environments often requires policies to integrate and maintain historical information in order to adapt within a task and generalize to new environments ( 15 ; 11 ; 5 ; 31 ) . Recurrent policies are well suited for these settings: they maintain internal hidden states that integrate past observations to support decision-making ( 6 ; 13 ; 11 ) . Despite their ubiquity and empirical success, the underlying properties of recurrent policies and the reasons for their superior performance remain poorly understood ( 19 ; 12 ) . Their decision-making processes are obscured by the complexity of the agent-environment interaction loop ( 15 ; 11 ) , the evolution of hidden state patterns ( 1 ; 14 ) , as well as diverse recurrent model architectures and policy learning methods ( 6 ; 13 ; 3 ; 24 ; 23 ) , developing a unified mechanistic understanding of recurrent neural policies has proven to be difficult. Until today, suitable explanatory analysis tools remain lacking for current neural policies. Methods such as feature attributions or saliency maps are typically designed to explain short-horizon, step-level effects, primarily used for rationalizing individual actions rather than coherent strategies ( 19 ; 12 ; 26 ; 28 ; 8 ; 34 ) . Conversely, while dynamical analyses can reveal meaningful low-dimensional structures in recurrent networks ( 29 ) , such analyses are rarely tied to policy execution in closed-loop environment interactions ( 18 ) . In this study, we find that surprisingly universal cyclic dynamical structures consistently emerge in the hidden state domain of fully optimized recurrent policies during their interaction with the environment. Such patterns hold across different task families, training pipelines (gradient-based policy optimization ( 24 ) and evolution strategies ( 23 ) ), and recurrent model architectures (conventional RNN structure and modern state space models like Mamba ( 9 ) ). Importantly, these cyclic patterns emerge in the policy’s hidden state even when memory is preserved across episode boundaries, indicating that they are a property of the learned dynamics rather than an artifact of state re-initialization. We further find that such cyclic structures can be well captured if viewing the agent execution as evolution in the state space of a hybrid dynamical system (HDS) whose joint state includes both the policy’s hidden neural states and environment states, with episode resets acting as discrete events ( 7 ; 2 ; 17 ) . In this view, the observed cyclic structures correspond to attracting limit cycles in the joint state space of HDS ( 27 ; 10 ) . These cycles serve as basic elements for orchestrating decisions over time: they stabilize the coupled evolution of policy’s memory and task-relevant environment state, and are robust to external perturbations, effectively suppressing nuisance variation from unknown environments. Beyond the existence of limit cycles, we also uncover a structured correspondence between limit cycles and policy behaviors. Different limit cycles correspond to distinct behavioral regimes, and the relationships among behaviors are mirrored by the geometry of the corresponding limit cycles. In particular, similarities between behaviors are strongly correlated with the geometric similarities of their limit cycles. This indicates that recurrent policies do not merely store information, but organize it into semantically meaningful representations that preserve relational structure at the level of behaviors. This might partly explain why recurrent policies often have better task generalization capabilities in meta-RL tasks ( 5 ; 31 ) . Finally, our findings resonate with recent breakthroughs in biological motor control, which reveal that neural latent dynamics are conserved across distinct individuals performing identical tasks ( 22 ) . This convergence suggests a fundamental principle shared by biological and artificial intelligence: neural manifolds are geometrically constrained to scaffold consistent physical behaviors. 2 Background 2.1 Sequential Decision Making with Recurrent Policies We study sequential decision-making problems in which policy learning requires integrating information over time, either because observations are partial (POMDP-style) ( 15 ) or because the task itself is underspecified and must be inferred online (meta-RL-style) ( 5 ; 31 ) . A convenient unifying view is to assume there exists a latent task/context variable c that remains fixed for an extended interaction, together with an environment state s_{t} that evolves over time ( 21 ; 35 ) . The agent observes o_{t} generated from the environment state, selects an action a_{t} , and receives reward r_{t} , while the environment dynamics depend on both the action and the fixed context ( 15 ; 21 ) . Meta-RL is naturally captured by a trial structure: the same context c is revisited over multiple episodes, where episode termination resets the environment state (e.g., s_{t} ) but keeps the context unchanged ( 5 ; 31 ; 21 ; 30 ) . Crucially, the agent is allowed to preserve internal memory across these episode boundaries, enabling within-trial adaptation without parameter updates. Recurrent policy is the default implementation choice for this requirement by maintaining an internal hidden state h_{t}\in\mathcal{H} that summarizes past interaction and is updated online ( 6 ; 13 ) : h_{t+1}=f_{\theta}(h_{t},o_{t}),\ a_{t}\sim\pi_{\theta}(\cdot\mid o_{t},h_{t})\ \text{or}\ a_{t}=\pi_{\theta}(o_{t},h_{t}). (1) This design is widely used in both POMDP and meta-RL tasks, where carrying h_{t} across time (and, in the trial setting, across episodes) supports online adaptation and empirically strong generalization to new task instances ( 11 ; 5 ; 31 ) . The central question of this paper is not whether recurrent policies can store historical information, but how they organize and integrate moment-to-moment decisions into coherent long-horizon strategies, and what execution-scale structure underlies their generalization and adaptation. 2.2 Preliminaries for Dynamical System Analysis Hybrid dynamical system (HDS). To conduct our investigation, we borrow basic concepts from dynamical system analysis ( 27 ; 7 ) . The key step is to analyze the coupled interaction between agent and environment, rather than the policy in isolation ( 15 ) . We treat all variables that evolve during execution on equal footing as a joint state x_{t}=(s_{t},h_{t}) , where s_{t} denotes the task/environment states and h_{t} the policy’s hidden neural state. The rollout is then modeled as a discrete-time hybrid dynamical system governed by the interplay between the agent’s internal dynamics F_{\theta} (memory update) and the environment’s transition dynamics G . Crucially, the system is hybrid because the environment dynamics G define a Jump Set \mathcal{D}\subset\mathcal{S} (e.g., goal locations or termination states) that triggers discrete events. The joint evolution \Phi_{\theta} switches behavior based on whether the physical state enters this set: x_{t+1}=\Phi_{\theta}(x_{t})=\begin{cases}\mathcal{T}_{\text{flow}}(x_{t}),&s_{t}\notin\mathcal{D},\\ \mathcal{T}_{\text{jump}}(x_{t}),&s_{t}\in\mathcal{D},\end{cases} (2) where \mathcal{T}_{\text{flow}} represents the nominal coupled evolution (driven by F_{\theta} and standard transitions in G ), and \mathcal{T}_{\text{jump}} represents the episodic reset dynamics. In the context of Meta-RL, this jump map \mathcal{T}_{\text{jump}} is distinctively structured: it re-initializes the physical state s_{t} while preserving the policy’s hidden memory h_{t} ( 5 ; 31 ) , thereby allowing the agent dynamics F_{\theta} to accumulate history and enable adaptation across episodes. (a) Unified State Space (b) Example of a Limit Cycle Figure 1 : Illustration of the Hybrid Dynamical System (HDS) framework. (a) Unified State Space. The execution state x_{t}=(s_{t},h_{t}) couples physical coordinates (orange axes, \mathcal{S} ) and neural hidden states (purple axes, \mathcal{H} ) into a single high-dimensional product space. (b) Example of a Limit Cycle. The system converges to a stable periodic orbit \Gamma . The vector field visualizes the basin of attraction, showing how trajectories from diverse initial conditions rapidly contract onto this cycle. This modeling choice is useful because it turns long-horizon policy execution into a state-space object: once the interaction is expressed as an HDS on x_{t} , we can leverage established dynamical system analysis tools to characterize how trajectories are organized, and in particular to search for stable structures that can serve as building blocks for long-horizon strategies ( 27 ) . Stable structures and limit cycles. Dynamical systems often use stable structures in state space to describe long-run behavior, such as fixed points, attracting manifolds (continuous attractors), and limit cycles ( 27 ; 10 ) . An attractor is a set toward which trajectories initialized nearby converge under repeated application of \Phi_{\theta} ( 27 ) . In our setting, attracting limit cycles are particularly relevant: a limit cycle is an isolated periodic orbit that is stable to small perturbations transverse to the orbit, meaning executions return to the same cycle after disturbances ( 27 ; 10 ) . Topologically, an attracting limit cycle is equivalent to a circle, which induces a natural phase coordinate along the orbit; this phase can persist over long time scales and provides a robust substrate for organizing long-horizon behavior ( 27 ) . To quantify system stability, a commonly used tool in dynamical system analysis is the K -step finite-time Lyapunov indicator (FTLI) ( 33 ; 25 ) . It can be computed by simulating paired executions : a nominal trajectory \{x_{t}\} and a “shadow” trajectory \{\tilde{x}_{t}\} initialized with a tiny controlled perturbation at time t_{0} , and tracking the evolution of their separation \delta_{t}:=\|x_{t}-\tilde{x}_{t}\| using a normalized norm. Specifically, we define \lambda_{FTLI} as the average exponential rate of divergence between these trajectories over the horizon K : \lambda_{FTLI}(t,K)=\frac{1}{K-1}\sum_{k=0}^{K-1}\log\frac{\delta_{t+k+1}}{\delta_{t+k}} (3) Crucially, the value of \lambda_{FTLI} reveals the local geometric structure of the system ( 33 ; 27 ) : • Convergence ( \lambda<0 ): The shadow trajectory is pulled back toward the nominal path (contraction), which is the defining signature of limit cycles. • Divergence ( \lambda>0 ): The trajectories drift apart exponentially, indicating sensitivity to initial conditions (chaos). • Neutrality ( \lambda\approx 0 ): Deviations persist without significant growth or decay, indicating a marginally stable or critical state between ordered and chaotic dynamics. Figure 2 : Overview of the experimental framework. We systematically evaluate recurrent policy dynamics across three axes of variation: (1) Task Families , ranging from partially observed grid maze navigation to high-dimensional Procgen games; (2) Training Pipelines , comparing gradient-based policy optimization method (PPO) against gradient-free evolution strategies; and (3) Recurrent Architectures , spanning classic RNNs, gated recurrent units (GRUs), and modern state-space model architecture Mamba. 3 Experimental Design To unravel the underlying working mechanisms of recurrent neural policies, we carefully design a systematic experimental framework to train recurrent policies over a diverse span of tasks, training schemes, and recurrent architectures. We are particularly interested in analyzing fully optimized recurrent policies that exhibit generalization capability. To achieve this, we adopt an RL 2 -style training protocol ( 5 ) : for each trial, we sample a task instance (context) and run multiple episodes under that fixed task, where episode termination resets the environment state, but the policy retains its internal memory across episode boundaries. We train agents to maximize return aggregated over the full trial, so that adaptation must be implemented through recurrent state dynamics rather than parameter updates. The full training details are provided in the Appendix A.1 . Specifically, we vary three axes of the learning system: • Task family. We evaluate both a partially observed grid maze navigation suite (procedurally generated mazes with varying origins and destinations) and several procedurally generated game-like environments from Procgen (e.g., CoinRun, Heist, Jumper) ( 4 ) , which differ in observation structure, dynamics, and long-horizon goals. Solving these tasks requires the policy to effectively integrate information over extended temporal horizons and exhibit strong generalization capability. • Training pipeline. We train policies using both gradient-free evolution strategies (ES) ( 23 ) and the gradient-based method PPO ( 24 ) , which allows testing if universal patterns emerge under drastically different policy optimization procedures. • Recurrent architecture. We implement recurrent policies with vanilla RNNs, GRUs, and modern state-space architecture Mamba ( 6 ; 3 ; 9 ) , in order to identify fundamental mechanisms that are architecture-agnostic. This setup allows us to carefully isolate impacts from task characteristics, policy optimization schemes, and recurrent architectures, thereby helping to locate the fundamental and universal mechanisms that contribute to the superior generalization and robustness performance of recurrent policies. 4 Emergence of Limit Cycles 4.1 Recurrent Policies Stabilize into Limit Cycles With the fully optimized recurrent policies, we test these policies’ behavior when solving new task instances in the corresponding environment. Again, we follow the episodic meta-RL setting, that let the agent continously solving the corresponding task through a series of trials. Although the physical environment state resets at episode boundaries, the agent’s internal memory (hidden neural states) persists, enabling adaptation across episodes. Typically, with a few trials, the optimized recurrent agents start to be able to solve the unseen tasks and gradually stabilize into convergent behaviors. We collect the hidden neural state sequences of the recurrent policies in such stabilized stages, and visualize them by performing dimension reduction using principal component analysis (PCA), as illustrated in Figure 3 (a). Interestingly, we observe that across diverse task settings (ranging from maze navigation to high-dimensional Procgen games), training pipeline (ES and PPO), and different recurrent architectures (RNN, GRU, Mamba), adapted recurrent agents always settle into stable periodic loopy structures. Some example loopy structures are presented in Figure 3 (a), more can be found in Appendix E . Once the policy has identified the right task context through the hidden state update, the coupled agent–environment system executes a repetitive behavioral pattern. Importantly, this emergent periodicity displays two defining dynamical properties: • Emergent neural loops: Although the hidden state h_{t} is not externally reset at episode boundaries, the neural trajectory spontaneously converges to a closed loop in low-dimensional projections. As shown in Figure 3 (a), the neural state traces a consistent periodic orbit that persists across later episodes, maintaining a cyclic pattern through the policy’s internal dynamics. • Stability under perturbation: The coupled system also exhibits stability under external disturbances. As depicted in Figure 4 , when either the physical environment state or the internal neural state is perturbed, the resulting neural loop only deviates transiently from the stable structure and then quickly recovers. Figure 3 : Task-adapted policies stabilize into limit cycles. (a) In the stable regime, the physical trajectory forms a topological loop enabled by episodic resets, while neural memory traces a low-dimensional closed loop (PCA projection). (b) Across task families and instances, the recurrent hidden-state trajectory converges to a stable closed orbit with task-dependent shape. Figure 4 : Robustness and recovery under external perturbations. The cycle acts as an attractor: after an external perturbation, execution transiently deviates but rapidly re-synchronizes with the nominal limit cycle. Additional experimental results on perturbation are provided in Appendix F . These properties—periodicity and convergence following perturbation— share remarkable similarities in both topology and dynamics properties with the concept of limit cycles in the hybrid dynamical systems (HDS) defined on the joint state x_{t}=(s_{t},h_{t}) of the environment state s_{t} and hidden neural state h_{t} , as introduced in Section 2.2 . To concretely verify whether such loopy structures are indeed limit cycles, we evaluate the local stability of these structures by calculating the Time Lyapunov Indicator (FTLI) using the method described in Section 2.2 . The FTLI distribution of fully optimized recurrent policies computed along the converged regime consistently concentrates below zero across diverse tasks, training methods, and recurrent architectures, as illustrated in Figure 5 . A negatively centered FTLI distribution indicates volume contraction and asymptotic stability. In contrast, randomly initialized networks evaluated under the same protocol exhibit positive exponents, indicating divergence. The recurrence of such cyclic topological structures implies that they are stable outcomes after policy learning. The attracting nature of the limit cycle reduces sensitivity to variability arising from environmental uncertainty, maintaining hidden neural state coherence in test scenarios, thereby offering recurrent policies better generalization and robustness performance as compared to non-recurrent policies. (a) Maze-ES-GRU (b) Jumper-PPO-GRU (c) Jumper-PPO-Mamba Figure 5 : Universality of contractive dynamics across architectures, tasks, and algorithms. FTLI distributions (computed with a horizon K=1000 ) for paired rollouts under small perturbations across three settings: (a) GRU/ES on Mazes; (b) GRU/PPO on Jumper; (c) Mamba/PPO on Jumper. Optimized policies (blue) consistently show \lambda_{FTLI}<0 (contraction), while random networks (yellow) show \lambda_{FTLI}>0 (chaos). Figure 6 : Periodically-Kicked Drive (PKD) mechanism. (Left) Episodic resets creates a quasi-periodic Observation Sequence . (Middle) These observations act as a Periodically Kicked Drive on the recurrent dynamics. (Right) Due to the network’s dissipative nature, this rhythmic forcing entrains the hidden state into a stable limit cycle . 4.2 Why Limit Cycles Arise: Episodic Task Structure Supports Periodically-Kicked Drive Although the periodicity of the environment state can be enforced by episodic resets, the neural state preserves memory across episodes, yet still converges to stable closed loops. This suggests that such neural periodicity is more likely a result of the interplay between the policy’s intrinsic dynamics and environment inputs. To explain this phenomenon, we find that the Periodically-Kicked Drive (PKD) mechanism ( 32 ) in dynamical system analysis might be an ideal tool to serve this purpose. PKD is widely used in dynamical system analysis to explain the formation of stable structures (such as limit cycles). As visualized in Figure 6 (left), the episodic reset structure guides the agent to encounter the same task conditions repeatedly. This encourages the agent to generate consistent action sequences, guiding the stabilization of the agent’s behavior, eventually producing a quasi-periodic stream of observations. This observation stream functions as a periodic external “kicked drive” acting on the recurrent network (Figure 6 (middle)). Dynamical systems analysis suggests that applying a PKD to a dissipative system entrains the system’s state into a stable limit cycle. We refer the interested readers to Theorem D.6 in Appendix D for more in-depth discussion and the theoretical connection between PKD and limit cycle formation in HDS. The recurrent policy exhibits the required dissipative properties: local linearization analysis confirms that the operational orbit lies within a strictly contractive region of the phase space (Appendix C ). This local evidence, combined with the negatively centered FTLI distribution (Figure 5 ), indicates that the periodic forcing provided by the observation sequence effectively entrains the hidden state into a stable limit cycle (Figure 6 (right)). Crucially, the existence of these limit cycles is not a trivial result of the environment reset mechanism. While resets provide the opportunity for periodicity, they do not guarantee the formation of a self-sustaining cycle. For a limit cycle to persist in the HDS, the neural orbit must generate the exact sequence of actions that reproduces its own driving observations. We utilized this constraint to empirically verify the limit cycle hypothesis. By employing the PKD mechanism as a probe on large-scale randomly initialized hidden neural states ( N=100,000 ), we successfully isolated stable orbits. As detailed in Appendix G and visualized in Figure 15 , only a specific subset of these entrained orbits satisfy the causal consistency check. The successful extraction of these high-precision closed loops—which exist only for trained networks and not for random baselines—provides constructive evidence that the observed structures are functional limit cycles inherent to the learned dynamics, rather than artifacts of environmental resets. 5 Structural Correspondence Between Behavior and Neural Dynamics Simply identifying the limit cycle structures hidden in recurrent policies only told half the story. We are more interested in understanding the meaning of these limit cycles: what do they represent, and how do they shape the physical behaviors of recurrent agents in the environment? Figure 7 : Structural isomorphism between behavior and neural dynamics. (a) A behavioral trajectory B^{*} (represented as a Behavioral Potential Field) maps to a unique Neural Limit Cycle. (b) CCA alignment of the behavioral manifold (top) and neural dynamics manifold (bottom), visualized with the first two canonical dimensions (CM). Colored arrows highlight the correspondence between specific behavioral fields (BPF1–4) and neural attractors (LC1–4), showing that topological relationships are preserved across the mapping. To investigate this, we generate a large-scale dataset of paired samples of agent behaviors and their corresponding limit cycles. Specifically, we use the PKD mechanism described in the previous Section, by replaying observation sequences from diverse behaviors, we drive the recurrent agent into stabilized periodic regimes. To isolate causal structures, we apply an Action-Consistency Filter (see Appendix G ) that retains only those neural cycles capable of reproducing the action sequence that generated their driving observations. This rigorous selection process yields a comprehensive library of validated limit cycles. Moreover, to model the highly diverse agent behaviors into comparable representations, we represent the agent behaviors using the Behavioral Potential Field (BPF). Taking the maze navigation task as an example, as shown in Figure 7 (a), we embed the variable-length agent trajectories into fixed-dimensional scalar fields where intensity decays with distance from the trajectory. The Euclidean distance between two BPFs approximates the transport cost between trajectories, providing a metric for behavioral similarity. For the limit cycle neural dynamics representation, we compute the centroid of each limit cycle in the neural hidden state space. Figure 8 : Correlated geometric correspondence between behavioral manifolds and neural limit cycles. CCA projections across diverse experimental configurations: GRU/ES/Maze, Mamba/PPO/Jumper, and GRU/PPO/Jumper, visualized with the first three canonical dimensions. Each figure is plotted based on 20,000 samples per task setting. (a) Color represents the distance of agent trajectories; (b) Color represents the orientation angle of agent trajectories. We measure the alignment between the above behavior and the neural dynamics representation using Canonical Correlation Analysis (CCA) ( 20 ) , a widely used technique in analyzing correlation structure between two datasets in multivariate statistics (see Appendix I for details). When plotting the mapped canonical variates of the behavior-neural representations using CCA, we observe a remarkable structural isomorphism between these two domains. As illustrated in Figure 7 (b), for the maze navigation task, its behavior manifold (top figure) shares almost the same geometry with the neural dynamics manifold of limit cycles (bottom figure). Moreover, such correspondence even carries over to the semantic meaning of behaviors. For example, the inner regions of both the behavior and the neural dynamics manifolds contain mostly short-distance trajectories, whereas the outer regions contain more long-distance trajectories. Even more surprisingly, as shown in Figure 7 (b), the residing locations of behaviors BF1-4 with different trajectory lengths and orientations in the behavior manifold are almost identical to the residing locations of their limit cycles in the neural dynamics manifold! Such a high-level correspondence not only exists in the maze navigation task, but also exists widely for recurrent policies trained in other task families, training methods, and model architectures, as shown in Figure 8 in the main text and more results in Appendix J . In these additional results, we also find a highly correlated geometric correspondence between behavioral and neural dynamics manifolds. Such a correspondence again carries rich and relational behavioral semantics, not only encoding the distance of agent trajectories, but also other behavioral attributes like the orientation angle of trajectories. This confirms that the neural dynamics topology preserves the metric properties of the physical behavior. All these results demonstrate a high level of structural isomorphism, that the manifold of neural limit cycles preserves the relational geometry of the physical behaviors, mapping similar trajectories to topologically adjacent attractors . This might partly explain why recurrent policies often have better task generalization capabilities in meta-RL tasks ( 5 ; 31 ) . As the recurrent agent’s underlying neural dynamics can encode continuous and relational structures of behaviors, it can smoothly adjust its strategy by traversing a continuous neural dynamics manifold, allowing the policy to interpolate the structured geometry to address novel task instances or non-stationary environments. Figure 9 : Quantitative universality of high-dimensional alignment. Canonical correlation spectra for the top 10 modes. Optimized agents (solid lines) sustain correlations >0.7 across 10 dimensions, whereas a randomized baseline (dashed line) decays rapidly, indicating that the alignment is a learned property. To further confirm the strength of canonical correlation between the behavior and neural dynamics domains, we expand our CCA-based analysis to higher canonical dimensions, instead of only inspecting the visualized patterns on the first 2-3 canonical dimensions. The results are presented in Figure 9 . Across all experimental settings—including Maze navigation and Procgen tasks trained via PPO or ES, using both GRU and Mamba—the canonical correlations between behavioral structure and neural dynamics remain high ( >0.7 ) for all 10 dimensions. To verify that this high-dimensional alignment is not an artifact of the state space dimensionality or action constraints, we conducted a random control experiment (see Appendix K for details). A randomized network, constrained to output the exact same action sequences ( N=10,000 samples), exhibits a spectral collapse after the first 3 dimensions. This contrast indicates that the observed 10+ dimensional alignment is a specific result of the learned recurrent dynamics. Moreover, we conduct counterfactual injection tests (see Appendix M ) to rigorously verify that this behavioral-neural dynamics alignment is causal, rather than epiphenomenal. By surgically manipulating the agent’s internal state—projecting optimal limit-cycle states into the canonical space and selectively lesioning specific dimensions—we test the functional role of the aligned geometry. As detailed in the convergence time histograms in Figure 23 (Appendix), we demonstrate that the top CCA-aligned dimensions are both necessary and sufficient for optimal control. Specifically, preserving only the top correlated modes (while randomizing all other neural dimensions) enables the agent to bypass exploration and immediately execute the optimal trajectory, shifting the convergence peak from \sim 250 to \sim 50 steps. Conversely, selectively randomizing just these top dimensions, even while keeping the vast majority of the neural state intact, completely abolishes this advantage, reverting the agent to baseline exploration performance. This confirms that the navigation-critical information is compressed almost entirely within the high-CCA subspace, establishing the aligned geometry as the functional basis of the agent’s adaptive behavior. 6 Conclusion We demonstrate that optimized recurrent neural policies in episodic settings consistently converge to attracting limit cycles. We identify Periodically-Kicked Drive (PKD) as a potential explanation for this phenomenon. Crucially, we reveal that the intrinsic geometry of these neural limit cycles preserves the relational structure information of the physical behavior, which holds regardless of the task family, training methods, and recurrent architectures. These establish the neural dynamics manifold of limit cycles as the functional basis of adaptive behavior. Our findings offer new perspectives to explain many nice properties of recurrent neural policies, such as their superior robustness and adaptation/generalization capabilities. 7 Discussion 7.1 Biological Connections and Geometric Universality Our results are consistent with recent findings in biological motor control. Safaie et al. ( 22 ) reported that neural latent dynamics are preserved across distinct individuals (monkeys and mice) performing identical motor tasks, despite variations in individual neural circuitry. While that study hypothesized that this preservation arises from evolutionary constraints on circuit development, our findings suggest a complementary explanation based on geometric constraints. We propose that the observed cross-individual alignment is an emergent property of solving a shared physical task. Since the physical environment imposes a specific relational geometry on the space of feasible behaviors, and optimized neural manifolds structurally mirror this behavioral geometry (structural isomorphism), the neural manifolds of distinct agents optimizing for the same task tend to converge to a similar geometric structure. Consequently, the ”shared neural landscape” observed by 22 may naturally arise whenever recurrent systems—biological or artificial—optimize policies for the same physical reality. 7.2 Limitations and Future Directions While this work establishes a correspondence between neural limit cycles and behavioral motifs, several aspects warrant further investigation. First, our analysis focused on episodic tasks where recurrent policies converge to limit cycles. However, not all Reinforcement Learning (RL) settings induce explicit periodicity. In non-episodic or continuous control tasks, recurrent policies might encode optimal strategies via different dynamical regimes, such as stable fixed points or chaotic attractors. We hypothesize that the principle of structural isomorphism extends to these alternative topological structures, though verification in non-periodic settings remains to be performed. Second, the Behavioral Potential Field (BPF) metric was implemented here primarily for 2D navigation tasks. Theoretically, BPFs can be generalized to high-dimensional state spaces, such as the joint-space trajectories of robotic manipulators. However, computing BPFs in high-dimensional spaces introduces significant computational scalability challenges. Developing efficient representations to quantify behavioral similarity in high-dimensional continuous control domains is a necessary step for generalizing these findings to complex robotic systems. Finally, while this study characterizes the stable limit cycles representing the converged policy, the transient dynamics of the optimization process remain to be explored. A critical next step is to analyze how the recurrent policy utilizes reward signals to drive the neural state from initialization toward these stable attractors. Applying dynamical systems tools to these transient trajectories could help elucidate the implicit ”in-context” optimization algorithms implemented by the RNN, which are meta-learned from data and may differ from hand-engineered update rules. Appendix A Experiment Setup To evaluate the universality of the observed dynamical structures, we conduct experiments across two distinct regimes: (1) a controlled, open-ended grid world trained via neuro-evolution, and (2) high-dimensional visual control tasks trained via gradient-based reinforcement learning. A.1 Open-Ended Maze Navigation (Evolution Strategies) Task Environment. We implement an open-ended navigation task on a 10\times 10 discrete grid. To ensure diversity, mazes are procedurally generated by assigning obstacles to cells with a 30% probability, followed by a connectivity check (via connected component analysis using OpenCV) to discard unsolvable layouts. This results in a large combinatorial search space ( 2^{100} potential configurations), preventing the agent from relying on memorization. The agent operates under partial observability, receiving input only from a 3\times 3 local window centered on its current position. This information bottleneck requires the agent to integrate information over time to build an internal model of the maze structure. The action space consists of four discrete movements: up, down, left, right . Recurrent Architecture. The agent is parameterized by a Gated Recurrent Unit (GRU). At each time step t , the network receives a concatenated input vector [o_{t},h_{t-1}] , where o_{t} is the flattened local perception and h_{t-1} is the hidden state from the previous step. A linear readout layer maps the updated hidden state h_{t} to a softmax distribution over the four actions. Meta-RL Trial Structure. We adopt an RL^{2} -style trial protocol to foster rapid adaptation. A single trial consists of multiple episodes within the same fixed maze. Crucially, when the agent reaches the goal (or times out), its physical position is reset to the start, but its hidden state is preserved . This continuity of memory allows the agent to exploit information gathered in early episodes to optimize trajectory efficiency in later episodes. Training via Evolution Strategies. We optimize the policy using OpenAI Evolution Strategies. The objective is to minimize the path length of the final episode in a trial, incentivizing the agent solve the maze. The fitness evaluation logic is detailed in Algorithm 1 . Algorithm 1 Meta-Training Fitness Evaluation 1: Input: Population of perturbed parameters \{\theta_{i}\}_{i=1}^{N} , Maze Generator \mathcal{M} 2: for each candidate \theta_{i} in parallel do 3: Sample maze M\sim\mathcal{M} 4: Initialize hidden state h_{0}=\mathbf{0} , position p_{0}=\text{Start} 5: // Execute Trial (Sequence of Episodes) 6: while Trial Budget not exceeded do 7: Run episode until Goal or Timeout 8: if Goal Reached then 9: L_{last}\leftarrow Current episode path length 10: Reset p\to\text{Start} only (keep h ) 11: else 12: Reset p\to\text{Start} and h\to\mathbf{0} (soft failure reset) 13: end if 14: end while 15: Return Fitness F_{i}=\text{CentralRankTransform}(L_{last}) 16: // Higher rank assigned to shorter final path lengths 17: end for A.2 High-Dimensional Visual Control (Procgen) Task Environment. We evaluate our hypothesis on the Procgen Benchmark, a suite of procedurally generated arcade-style environments that feature high-dimensional visual observations ( 64\times 64\times 3 RGB) and diverse level structures. We specifically focus on tasks such as CoinRun , Jumper , and Heist , which require the agent to navigate complex spatial layouts and retain memory of key environmental features. To test generalization, we use a fixed set of seeds for training levels and a disjoint set of seeds for evaluation levels. Recurrent Architecture. The agent architecture consists of a convolutional encoder followed by a recurrent dynamics model. The encoder processes the visual input into a compact latent embedding. This embedding is then fed into the recurrent core—either a standard Gated Recurrent Unit (GRU) or a Mamba-based State Space Model—which maintains the agent’s hidden state h_{t} . The recurrent output is projected by two separate linear heads: a policy head producing a categorical distribution over discrete actions, and a value head estimating the state value function. Meta-RL Trial Structure. Consistent with the RL^{2} framework, we organize training into “trials.” Each trial consists of a fixed number of episodes (e.g., K=2\sim 5 ) played on the same unique level seed. Crucially, the recurrent hidden state is preserved across episode boundaries within a trial, and is only reset to zero when a new trial (i.e., a new level seed) begins. This setup forces the agent to use its activation dynamics to adapt to the specific level layout over the course of the trial. Training via PPO. We train the recurrent policies using Proximal Policy Optimization (PPO) with Generalized Advantage Estimation (GAE). We use the Adam optimizer with a learning rate of 5\times 10^{-4} . The training process runs on parallel environments to collect experience, with gradients averaged across mini-batches. To encourage exploration and prevent premature convergence, we include an entropy regularization term in the objective. The optimization loop allows the agent to continuously refine its internal dynamics to maximize the cumulative reward over the entire trial duration. Appendix B Hybrid Dynamical Systems System Modeling. To understand the causal coupling between the agent and the environment, we formalize the interaction as a closed-loop Hybrid Dynamical System (HDS) . We define the joint state space as \mathcal{X}=\mathcal{S}\times\mathcal{H} , where \mathcal{S} represents the discrete environmental states (e.g., maze coordinates, obstacle configurations) and \mathcal{H}\subseteq\mathbb{R}^{K} denotes the continuous neural hidden states. The time-evolution of this joint system can be condensed into two coupled recurrence relations representing the agent and the environment respectively: \begin{cases}h_{t+1}=f_{\theta}(\mathrm{obs}(s_{t}),h_{t}),&\text{(Agent Memory Dynamics)}\\ s_{t+1}=\mathcal{T}(s_{t},\pi_{\theta}(s_{t},h_{t})).&\text{(Physical State Evolution)}\end{cases} Here, the first equation governs the agent’s internal evolution, where the recurrent function f_{\theta} updates the memory based on the partial observation derived from the current physical state s_{t} . The second equation governs the physical evolution, where the environment’s transition operator \mathcal{T} updates the physical state based on the action sampled directly from the policy \pi_{\theta}(s_{t},h_{t}) . This formulation establishes that the agent-environment loop constitutes a unified dynamical entity. By tracking the evolution of the joint state x_{t}=(s_{t},h_{t}) in the product space \mathcal{X} , we address the bidirectional coupling characteristic of embodied intelligence: we no longer view the network as a passive responder to environmental stimuli, but as an active component in a bidirectional causal loop where neural states drive physical transitions and vice versa. Appendix C Empirical Verification of Local Contractivity in Neural State Space Before formally deriving the Periodically-Kicked Drive (PKD) mechanism, we present empirical evidence verifying a necessary condition for limit cycle formation: the existence of a contractive region in the neural state space. We investigate whether the stable trajectories generated by a trained agent are confined within a region where the intrinsic neural dynamics exhibit volume contraction. C.1 Local Linear Dynamical System (LDS) Analysis To quantify the local stability characteristics along the neural trajectory, we analyze the agent’s recurrent dynamics as a continuous-time vector field. For a recurrent unit (e.g., RNN) with hidden state h\in\mathbb{R}^{n} , the intrinsic update dynamics (ignoring external inputs for local stability analysis) can be approximated by the vector field: \dot{h}\approx F(h)=\tanh(W_{rec}h+b)-h (4) where W_{rec} represents the recurrent weight matrix and b is the bias. An equilibrium or stable manifold requires the divergence of this field to be negative. We employ Local Linearization to analyze the stability at each point h_{t} along the recorded trajectory. By performing a first-order Taylor expansion around h_{t} , we approximate the non-linear system as a Local Linear Dynamical System (LDS): F(h)\approx F(h_{t})+J_{F}(h_{t})\cdot(h-h_{t}) (5) where J_{F}(h_{t})=\frac{\partial F}{\partial h}\big|_{h_{t}} is the Jacobian matrix evaluated at the current state. The stability of the system at point h_{t} is determined by the eigenspectrum of the Jacobian J_{F}(h_{t}) . Specifically, if the real parts of all eigenvalues \lambda_{i} are negative: \max_{i}\text{Re}(\lambda_{i}(J_{F}(h_{t})))<0 (6) then the local dynamics are contractive, meaning that small perturbations in the state space will decay over time, pulling the system back towards the attractor. C.2 Experimental Results We conducted this analysis on a GRU agent fully trained on a POMDP Grid Maze task. Once the agent stabilized into a repetitive optimal strategy, we captured a sequence of 50 neural states along its limit cycle. Figure 10 visualizes the results of this stability check: • Behavioral & Neural Cycles (Top): The agent performs a consistent loop in the maze (Top Left), which corresponds to a distinct closed orbit in the PCA projection of the neural states (Top Right). • Eigenvalue Spectra (Bottom): For every single time step ( T1 to T50 ) along this orbit, we computed the eigenvalues of the local Jacobian. The bar charts display the sorted real parts of these eigenvalues. Conclusion: As evidenced by the uniformly blue histograms, the maximum real eigenvalue is strictly negative for every point along the trajectory. This confirms that the entire operational orbit of the agent lies within a locally contractive region of the state space. This consistent contractivity provides the dynamical basis for the formation of stable limit cycles, as it ensures that the neural state is constantly being converging towards the attractor, counteracting any expansive effects from environmental noise or inputs. Figure 10 : Empirical verification of contractive dynamics along the neural limit cycle. (Top Left) The physical trajectory of a converged GRU agent in a POMDP Maze task. (Top Right) PCA visualization of the neural hidden states for the last 50 steps, revealing a low-dimensional closed orbit structure. (Bottom) Detailed eigenvalue spectra for the local Jacobian matrix at each of the 50 time steps along the cycle. The y-axis represents the real part of the eigenvalues. Across all time steps ( T1-T50 ), the spectral components are entirely negative (blue bars), with no positive components (which would appear as red bars above the dashed line). This indicates that the local dynamics are strictly contractive ( \text{Re}(\lambda)<0 ) throughout the entire execution of the behavior, providing the necessary dynamical condition for stable limit cycle maintenance. Appendix D Theoretical Analysis of Stability and Limit Cycles Conditioned Neural Subsystem. We analyze the stability of the recurrent policy by focusing on its neural subsystem, while conditioning on the effective input sequence induced by the closed-loop agent–environment interaction described in Section B . Specifically, the physical environment state s_{t} influences the recurrent dynamics only through the observation and encoding pipeline, yielding an effective input sequence \{u_{t}\} to the recurrent policy. Under this conditioned viewpoint , we provide a formal proof that a periodically driven, dissipative recurrent policy induces a unique, globally attracting stable limit cycle. D.1 Setup and Definitions We first formalize the properties of the periodic input drive and the dissipative nature of the policy’s recurrent dynamics. Definition D.1 (Periodic Input Drive) . Let \{u_{t}\}_{t\geq 0} denote the sequence of effective inputs to the recurrent policy, induced by the closed-loop agent–environment dynamics through the observation and input-encoding pipeline. We say the drive is T -periodic if: u_{t+T}=u_{t},\qquad\forall t\geq 0. (7) Definition D.2 (Contractive Region) . A subset \mathcal{C}\subset\mathbb{R}^{n} is called a contractive region if it is forward-invariant under the policy dynamics and the recurrent update is strictly contractive within \mathcal{C} . Specifically: 1. ( Forward invariance ) For any h\in\mathcal{C} and any admissible input u , we have f_{\theta}(u,h)\in\mathcal{C}. 2. ( Local contractivity ) There exists a norm \|\cdot\| and a constant 0<\lambda<1 such that for all h,h^{\prime}\in\mathcal{C} and any input u , \|f_{\theta}(u,h)-f_{\theta}(u,h^{\prime})\|\leq\lambda\|h-h^{\prime}\|. Definition D.3 (Dissipative Policy Dynamics (Restricted)) . Let h_{t}\in\mathbb{R}^{n} be the policy memory state evolving according to h_{t+1}=f_{\theta}(u_{t},h_{t}) . We assume that there exists a contractive region \mathcal{C}\subset\mathbb{R}^{n} such that the dynamics are strictly contractive within \mathcal{C} in the sense of Definition D.2 . To analyze the long-term behavior over full episodes, we introduce the stroboscopic map. Definition D.4 (Stroboscopic Map) . Fix a T -periodic effective input sequence \{u_{t}\}_{t=0}^{T-1} , induced by a converged closed-loop behavioral regime of the hybrid system. Let F_{t}(\cdot):=f(u_{t},\cdot) . The stroboscopic map S:\mathbb{R}^{n}\to\mathbb{R}^{n} represents the state evolution over one complete period T : S:=F_{T-1}\circ F_{T-2}\circ\cdots\circ F_{0}. (8) A fixed point h^{*} of S (i.e., S(h^{*})=h^{*} ) corresponds to the initial state of a T -periodic trajectory. We assume that the trajectory \{h_{t}\} remains entirely within the contractive region \mathcal{C} . D.2 Proof of Existence and Uniqueness Lemma D.5 (Contraction of the Stroboscopic Map) . Assume the policy dynamics are contractive within a region \mathcal{C} and that the trajectory remains in \mathcal{C} . Then the stroboscopic map S restricted to \mathcal{C} is a contraction mapping with Lipschitz constant \lambda^{T} . That is: \|S(h_{1})-S(h_{2})\|\leq\lambda^{T}\|h_{1}-h_{2}\|,\qquad\forall h_{1},h_{2}\in\mathcal{C}. (9) Proof. By Definition D.3 , each single-step map F_{t} is \lambda -Lipschitz. Since S is the composition of T such maps, the Lipschitz constant of S is the product of the individual constants. Thus, L_{S}\leq\prod_{t=0}^{T-1}\lambda=\lambda^{T} . Since 0<\lambda<1 , we have \lambda^{T}<1 . ∎ Theorem D.6 (Existence of a Stable Limit Cycle within a Contractive Region) . Given a T -periodic input and dissipative policy dynamics within a contractive region \mathcal{C} , there exists a unique T -periodic trajectory entirely contained in \mathcal{C} . Furthermore, for any initial state h_{0}\in\mathcal{C} , the policy trajectory converges exponentially to this periodic orbit. Proof. Existence and Uniqueness: Since S is a contraction mapping on the Banach space \mathbb{R}^{n} (Lemma 1), by the Banach Fixed-Point Theorem, S admits a unique fixed point h^{*} . This fixed point generates the periodic orbit h_{t+1}^{*}=f(u_{t},h_{t}^{*}) with h_{0}^{*}=h^{*} , satisfying h_{T}^{*}=S(h^{*})=h^{*} . Global Convergence: The Banach Fixed-Point Theorem further guarantees that for any initial h_{0} , the sequence of states at the start of each episode, h_{kT}=S^{k}(h_{0}) , converges to h^{*} as k\to\infty . specifically: \|h_{kT}-h^{*}\|\leq\lambda^{kT}\|h_{0}-h^{*}\|. (10) This implies exponential convergence of the dynamical system to the unique limit cycle. ∎ Appendix E Empirically Observed Limit Cycles Figure 11 provides a qualitative survey of empirically observed limit cycles in Procgen, spanning 8 environments, 3 recurrent architectures (RNN, GRU, Mamba), and 2 training pipelines (PPO, evolution strategies). Each subpanel corresponds to a fixed task–method–architecture configuration and contains three representative procedural seeds; for each seed we show the initial observation (top strip) and the resulting PCA-projected recurrent hidden-state trajectory over one converged episode (bottom). Across all settings, the projected trajectories exhibit consistent signatures of a stable periodic attractor in the closed-loop agent–environment dynamics: (i) a simple closed-orbit topology (clear loops rather than space-filling curves), (ii) smooth, non-self-intersecting paths indicative of evolution on a low-dimensional manifold, and (iii) periods commensurate with the episode length. Moreover, the high-dimensional closure residual \|\mathbf{h}_{T}-\mathbf{h}_{0}\|_{2} concentrates around \mathcal{O}(10^{-5}) , supporting that the observed loops reflect genuine periodic orbits rather than projection artifacts (Appendix G ). These examples are intended as qualitative support for the broader empirical regularity: after convergence, recurrent policies frequently settle into stable cyclic regimes, consistent with the theoretical prediction that dissipative dynamics under periodic forcing promote convergence to stable periodic orbits (Appendix D ). Figure 11 : Gallery of empirically observed limit cycles. Each subpanel corresponds to a fixed (task, training pipeline, architecture) configuration and shows three representative procedural seeds. For each seed, the top strip shows the initial observation and the bottom shows the PCA-projected recurrent hidden-state trajectory over a converged episode. Across environments and training setups, trajectories consistently form stable, smooth closed loops, with orbit geometry varying by task but remaining topologically simple. Figure 12 : Cross-architecture consistency under fixed environment and seed. We fix the Procgen environment and the procedural seed, and compare the converged recurrent dynamics across different architectures. As in Figure 11 , the top strip shows the seed’s initial observation and the bottom shows the PCA-projected hidden-state trajectory over an episode. Despite architectural differences, the resulting limit-cycle geometry is notably similar, suggesting that near-optimal closed-loop solutions induce a task-instance-conditioned attractor whose coarse geometric structure is largely architecture-invariant. This qualitative observation is consistent with our quantitative neural–behavioral alignment analysis (Appendix J ), which reveals high-dimensional correspondences between neural manifolds and behavioral geometry across settings. Appendix F Robustness of Limit Cycles under Hidden-State Perturbations To extend the illustrative example in Figure 4 , we evaluate limit-cycle stability under perturbations across many task instances (procedural seeds) and recurrent policies in both maze navigation and Procgen . For each trained policy \pi_{\theta} (parameters \theta fixed) and a fixed task instance, we first record a baseline hidden-state trajectory \{h_{t}^{\mathrm{base}}\}_{t=0}^{T} in the converged regime, then inject an additive perturbation at a chosen time step t^{*} : \tilde{h}_{t^{*}}=h_{t^{*}}^{\mathrm{base}}+\epsilon\,\delta,\qquad\delta\sim\mathcal{N}(0,I). (11) The perturbation is applied only to the recurrent hidden state; the environment dynamics and observation stream are otherwise unchanged. We run several perturbation variants per instance (independent draws of \delta ) and visualize baseline and perturbed trajectories in the same PCA projection used throughout the paper (PCs fitted on the baseline trajectory). Figure 13 : Perturbation robustness across maze navigation and Procgen. Each panel shows PCA-projected hidden-state trajectories for a trained policy on a fixed task instance. The darker curve is the unperturbed baseline cycle; lighter curves are trajectories after hidden-state perturbations (policy weights fixed). Across a broad set of environments and seeds, perturbations induce only transient deviations and trajectories contract back to the same task-conditioned limit cycle, consistent with the contractive dynamics in Theorem D.6 . Boundary cases We also observe counterexamples where perturbations lead to substantially longer transients before trajectories approach a stable cyclic regime. Figure 14 highlights two such Procgen environments, Bossfight and Bigfish . Compared to maze navigation, these games include richer non-stationary interaction dynamics (e.g., moving adversaries, projectiles, prey) whose evolution is not fully controlled by the agent and can dominate the visual stream; equivalently, the closed-loop system is driven by higher-dimensional exogenous degrees of freedom. Moreover, successful behavior in Bossfight and Bigfish is often primarily reactive—requiring continual adjustment to the instantaneous configuration of enemies/projectiles/fish—which further weakens the emergence of a short, repeatable observation–action pattern (periodic forcing) and yields longer excursions in hidden-state space. Nevertheless, partial re-stabilization is still visible for some seeds (e.g., Bossfight seed 34; Bigfish seed 122), suggesting that contractive basins may exist locally even when global re-convergence is slower. Figure 14 : Extended transients under perturbation in Procgen. In Bossfight and Bigfish , hidden-state perturbations can induce prolonged excursions before trajectories approach a stable regime. These environments exhibit stronger non-stationarity due to moving entities and reactive interactions, which can disrupt the quasi-periodic structure that supports rapid re-convergence in maze navigation. Partial stabilization is still visible for some seeds (e.g., Bossfight seed 34; Bigfish seed 122). Appendix G Limit Cycle Sampling via PKD and Action-Consistency Filter (ACF) While the Periodically-Kicked Drive (PKD) mechanism effectively reveals the distribution of potential limit cycles within the recurrent policy’s state space, not all such cycles are functionally relevant. A periodic input stream might entrain the network into a stable ”ghost orbit” that is mathematically stable under the forced input but generates actions inconsistent with the input itself. Such an orbit would break the causal loop in a real interaction. To isolate the specific neural structure responsible for a given behavior, we employ the Action-Consistency Filter (ACF) . This procedure verifies the closed-loop self-sustainment condition: a valid neural limit cycle must produce the exact sequence of actions required to generate the observation stream that drives it. The filtering pipeline proceeds as follows: 1. PKD Extraction: We first record a ground-truth trajectory of length T (the period) from the agent’s interaction with the environment, extracting the cyclic observation sequence \mathcal{O}_{seq} and the target action sequence \mathcal{A}_{target} . 2. Large-Scale Probing: We initialize a large population of random hidden states ( N=100,000 ) to broadly sample the basins of attraction in the neural state space. 3. Entrainment: We apply the observation sequence \mathcal{O}_{seq} as a repeated external drive to these states for a sufficient warmup period (e.g., 1,000 steps). Due to the contractive nature of the trained network, most trajectories will collapse onto stable manifolds. 4. Convergence & Consistency Check: After entrainment, we first discard trajectories that have not converged to a periodic orbit. To address concerns regarding high-dimensional closure, we explicitly measure the Closure Error \delta=\|\mathbf{h}_{T}-\mathbf{h}_{0}\|_{2} . We find that for retained candidates, \delta is concentrated at the magnitude of 10^{-5} , confirming that these are genuine periodic orbits in the high-dimensional space, not merely projection artifacts. 5. Filtering: Finally, we check the readout constraint: does the neural state at each step of the cycle decode to the correct action in \mathcal{A}_{target} ? Only cycles satisfying both convergence and action consistency are retained. The detailed algorithm is provided in Algorithm 2 . Algorithm 2 Action-Consistency Filter (ACF) 1: Input: Recurrent Policy f_{\theta} , Readout \pi_{\theta} 2: Input: Ground Truth Obs Sequence \mathcal{O}=\{o_{1},\dots,o_{T}\} , Target Actions \mathcal{A}=\{a_{1},\dots,a_{T}\} 3: Hyperparameters: Number of seeds N=100000 , Warmup steps K=1000 , Convergence thresh \epsilon 4: Output: Set of Validated Limit Cycles \mathbf{C}_{valid} 5: Initialize \mathbf{C}_{valid}\leftarrow\emptyset 6: Generate N random initial hidden states \{h^{(i)}_{0}\}_{i=1}^{N}\sim\mathcal{N}(0,I) 7: for each candidate state h^{(i)}_{0} in parallel do 8: h\leftarrow h^{(i)}_{0} 9: // Phase 1: Entrainment (Apply PKD) 10: for step k=1 to K do 11: t\leftarrow(k-1)\pmod{T}+1 12: h\leftarrow f_{\theta}(h,o_{t}) 13: end for 14: // Phase 2: Capture Cycle & Check Convergence 15: h_{start}\leftarrow h 16: \text{Cycle}\leftarrow[] 17: \text{Actions}\leftarrow[] 18: for step t=1 to T do 19: a_{pred}\leftarrow\arg\max\pi_{\theta}(h) 20: \text{Cycle.append}(h) 21: \text{Actions.append}(a_{pred}) 22: h\leftarrow f_{\theta}(h,o_{t}) 23: end for 24: // Check 1: Is it a closed loop? (Periodicity) 25: \delta_{closure}\leftarrow\|h-h_{start}\|_{2} 26: if \delta_{closure}>\epsilon then 27: continue 28: end if 29: // Check 2: Action Consistency 30: if \text{Actions}==\mathcal{A} then 31: \mathbf{C}_{valid}.\text{add}(\text{Cycle}) 32: end if 33: end for 34: Return \mathbf{C}_{valid} Figure 15 : Visualizing the landscape of entrained attractors under Periodically-Kicked Drive. We probe the neural state space of a GRU policy by initializing 100,000 random hidden states and driving them with a fixed, task-derived PKD sequence. The resulting converged orbits are projected via PCA. Color Coding: Orbits are colored by their Action Consistency Error —the discrepancy between the policy’s open-loop action outputs and the ground-truth actions required to sustain the cycle (Green: Low Error/Consistent; Red: High Error/Inconsistent). Insight: While the periodic drive entrains a vast number of states into stable periodic orbits (the dissipative ”ghost orbits” shown in red), only specific, tightly clustered regions (green) satisfy the causal self-sustainment condition. This visualizes that periodicity provided by resets/PKD is necessary but not sufficient; the learned internal structure must also align to close the sensorimotor loop. Distinguishing Learned Dynamics from Trivial Resets. The emergence of these self-sustaining limit cycles is not a trivial consequence of the episodic task structure. While environmental resets inherently impose periodicity on the observation stream, potentially entraining the neural state into a passive loop, our ACF analysis reveals that this external drive is insufficient for closed-loop stability. As visualized in Figure 15 , when we probe the state space with 100,000 initializations under a fixed PKD, we find a vast landscape of mathematically stable attractors. However, the majority of these are ”ghost orbits” (red) that fail the action-consistency check. They are merely entrained echoes of the input forcing. Valid limit cycles (green) appear only as distinct, isolated clusters where the policy’s internal dynamics precisely align with the external drive to produce the correct actions. In a real interaction, any state falling into the inconsistent region would immediately diverge from the cycle after a reset, breaking the loop. Thus, the limit cycles we observe are specific, learned dynamical structures, not artifacts of environmental periodicity. Appendix H Behavioral Embedding: Behavioral Potential Field (BPF) We introduce the Behavioral Potential Field (BPF) as a method to encode variable-length navigation paths into a fixed-resolution scalar field. This representation allows us to quantify geometric similarities—such as path shape, direction, and detour magnitude—using standard vector space metrics. Geometric Intuition and Validation. Why does the Euclidean distance between two scalar fields represent a meaningful metric for trajectory similarity? Conceptually, BPF transforms a sparse sequence of coordinates into a dense spatial distribution. When we compute the L_{2} distance between two BPFs, we are calculating the integrated difference in their potential intensities. This approximates a transport-cost metric (similar to the Wasserstein distance) without the computational overhead. To verify that this metric aligns with human intuition, we analyzed two distinct scenarios using synthetic trajectories, as shown in Figure 16 : • Spatial Separation (Figure 16 , Top): We compared three non-overlapping paths. Visually, trj1 is spatially closer to trj2 than to the distant trj3 . The BPF Euclidean distance captures this relationship accurately: d(\text{trj1},\text{trj2})=4.38 , which is significantly smaller than d(\text{trj1},\text{trj3})=7.35 . • Shape Deformation (Figure 16 , Bottom): For paths that overlap (sharing start/end points), the metric distinguishes between minor deviations and major detours. A slight deviation ( trj2 ) yields a distance of d=2.32 from the reference trj1 , while a larger detour ( trj3 ) yields d=4.67 . These results confirm that the BPF embedding preserves both the topological shape and spatial displacement of behaviors in a linear metric space, providing a robust basis for CCA alignment. Figure 16 : Validation of the BPF metric against geometric intuition. 1. Non-overlapping scenario (Top): The computed Euclidean distances in BPF space align with the visual spatial separation: d(\text{trj1},\text{trj2})=4.38