CACTO-BIC: Scalable Actor-Critic Learning via Biased Sampling and GPU-Accelerated Trajectory Optimization Elisa Alboni, Pietro Noah Crestaz, Elias Fontanari, Andrea Del Prete Abstract Trajectory Optimization (TO) and Reinforcement Learning (RL) offer complementary strengths for solving optimal control problems. TO efficiently computes locally optimal solutions but can struggle with non-convexity, while RL is more robust to non-convexity at the cost of significantly higher computational demands. CACTO (Continuous Actor-Critic with Trajectory Optimization) was introduced to combine these advantages by learning a warm-start policy that guides the TO solver towards low-cost trajectories. However, scalability remains a key limitation, as increasing system complexity significantly raises the computational cost of TO. This work introduces CACTO-BIC to address these challenges. CACTO-BIC improves data efficiency by biasing initial-state sampling leveraging a property of the value function associated with locally optimal policies; moreover, it reduces computation time by exploiting GPU acceleration. Empirical evaluations show improved sample efficiency and faster computation compared to CACTO. Comparisons with PPO demonstrate that our approach can achieve similar solutions in less time. Finally, experiments on the AlienGO quadruped robot demonstrate that CACTO-BIC can scale to high-dimensional systems and is suitable for real-time applications. I INTRODUCTION Trajectory Optimization (TO) is a widely used and flexible technique for solving robotic control problems. In TO, the high-level task is formulated as a constrained Optimal Control Problem (OCP), where the optimization variables are the system’s state and control trajectories. Constraints enforce compliance with system dynamics and kinematics, actuator limits, and task-specific requirements. However, OCPs are typically highly non-convex, making gradient-based solvers prone to converge to poor local minima. While global methods based on the Hamilton–Jacobi–Bellman equation or Dynamic Programming [ 5 ] exist, their applicability is limited by the curse of dimensionality. Deep Reinforcement Learning (RL) emerged as an alternative framework, particularly for continuous state and action spaces. Algorithms such as DDPG [ 25 ] , SAC [ 16 ] , and PPO [ 35 ] have demonstrated strong performance in robotic control tasks. Due to their exploratory nature, RL methods are generally less sensitive to local minima but they typically suffer from high sample complexity and long training times. To overcome the complementary limitations of TO and RL, hybrid approaches combining the two have recently gained significant research attention. A popular choice is to rely on TO imitation : policies are trained to mimic TO or model predictive control (MPC) solutions—through value-based or action-based imitation—to reduce online computational costs and to leverage sensor feedback [ 8 , 14 ] . However, these methods neither improve TO solution’s quality, nor guarantee constraint satisfaction. Accounting for policy approximation errors in the TO problem can lead to better results [ 22 , 24 ] , but it inherits the same limitations. Other methods use learned policies or value functions to warm-start TO or to define terminal costs, thereby accelerating optimization and guiding the solver toward improved solutions [ 32 , 9 ] . While effective for TO, these approaches provide no benefits for RL training. An increasingly prominent class of approaches embeds TO directly within the RL framework to improve training efficiency. These methods can be classified based on where TO is included: post-policy , pre-policy , or as a residual policy. Post-policy methods evaluate TO after the policy. Some methods learn cost or constraint parameters [ 33 , 44 ] , leveraging MPC’s safety and stability guarantees, but require solving TO online and face convergence challenges. Others use TO as actor and learn its terminal cost [ 26 , 20 ] , accelerating training and improving constraint handling, yet potentially yielding suboptimal solutions or neglecting sensor feedback. Another variant initializes TO with an RL policy [ 15 , 28 ] , improving convergence speed and solution quality, yet still does not exploit sensor feedback. In pre-policy methods, TO generates reference trajectories or auxiliary information that are inputted to the RL policy, effectively speeding up training [ 19 ] . Residual methods instead learn a residual policy to improve TO-generated control inputs with learned corrections [ 36 ] .While both pre-policy and residual methods accelerate training, they require online TO and depend on its ability to find high-quality solutions; moreover, learned policies may violate constraints even when TO does not. In this work, we extend CACTO [ 15 , 1 ] , a post-policy algorithm that exploits the interplay between TO and RL to accelerate training. The actor policy generates the initial guess for TO, leveraging the exploratory nature of RL to avoid convergence to poor local minima, while TO guides the learning process of the RL agent. The primary limitation of CACTO is scalability. As the system complexity increases, TO becomes more expensive, and actor–critic training requires more iterations to converge. We investigate strategies to reduce the computational burden in both TO and RL phases. Our main contributions are: • A method to identify state-space regions where an improvement of the actor policy is more likely. • A new version of CACTO’s algorithm, called CACTO-BIC, that exploits biased initial conditions (BIC) and GPU-based computation to achieve improved sample and time efficiency. • A JAX-based [ 6 ] open-source implementation of CACTO-BIC that exploits GPUs to solve TO problems and train neural networks. • The first validation of CACTO on real hardware through experiments on a quadruped robot. II BACKGROUND This section summarizes the latest formulation of CACTO (originally called CACTO-SL [ 1 ] ), an algorithm for solving finite-horizon discrete-time optimal control problems such as: \displaystyle\mathmakebox[\widthof{$\underset{\displaystyle\scriptstyle{X,U}}{\mathrm{subject~to}}$}][l]{\underset{\displaystyle\scriptstyle{X,U}}{\mathrm{minimize}}}\quad L(X,U)\triangleq\sum_{k=0}^{T-1}l_{k}(x_{k},u_{k})+l_{T}(x_{T}) \displaystyle\mathmakebox[\widthof{$\underset{\displaystyle\phantom{\scriptstyle{X,U}}}{\mathrm{subject~to}}$}][c]{{\mathrm{subject~to}}}\quad \displaystyle x_{k+1}=f_{k}(x_{k},u_{k}),\quad k=0,\dots,T-1, \displaystyle|u_{k}|\leq u_{\max},\quad\quad\quad\,\,\,\,k=0,\dots,T-1, \displaystyle x_{0}=x_{\mathrm{init}}, where X=\{x_{0},\dots,x_{T}\} and U=\{u_{0},\dots,u_{T-1}\} are the state and control sequences, with x_{k}\in\mathbb{R}^{n} and u_{k}\in\mathbb{R}^{m} . The cost L(\cdot) combines running costs l_{k} and a terminal cost l_{T} , while the constraints enforce system dynamics, control bounds, and initial conditions. CACTO begins by solving N TO problems from randomly sampled initial states, using standard warm-starting (e.g., setting all states to x_{\mathrm{init}} and controls to zero). As the OCP horizon is finite, the actor policy and the critic value are time dependent, so also the initial time is randomized in each TO instance and the time is appended to the state vector, \tilde{x}=[x,t] . For each state of each optimized trajectory, CACTO computes the partial K -step cost-to-go \bar{V} , where K is a user-defined parameter representing the Temporal Difference lookahead horizon. The value function’s gradient \bar{V}_{x} is also computed, using the backward pass of Differential Dynamic Programming (DDP) [ 18 ] . These values (\bar{V},\bar{V}_{x}) , together with the associated state, control, and state after K steps, are stored in a replay buffer. Afterwards, the critic and actor neural networks are trained for M iterations using mini-batches sampled from the replay buffer. The critic approximates the value function and its gradient by matching \bar{V} and \bar{V}_{x} : \displaystyle\mathmakebox[\widthof{$\underset{\displaystyle\scriptstyle{\theta^{V}}}{\mathrm{subject~to}}$}][l]{\underset{\displaystyle\scriptstyle{\theta^{V}}}{\mathrm{minimize}}}\quad(\bar{V}-V(\tilde{x}|\theta^{V}))^{2}+k_{s}(\bar{V}_{x}-S_{x}V_{\tilde{x}}(\tilde{x}|\theta^{V}))^{2} (1a) where V(\cdot) , V_{\tilde{x}}(\cdot) and \theta^{V} are the critic network, its gradient (with respect to \tilde{x} ) and its parameters, k_{s} is the weight of the gradient term, and S_{x} the selection matrix to exclude the partial derivative of V with respect to time. The actor is updated by minimizing the Q-value: \displaystyle\mathmakebox[\widthof{$\underset{\displaystyle\scriptstyle{\theta^{\mu}}}{\mathrm{subject~to}}$}][l]{\underset{\displaystyle\scriptstyle{\theta^{\mu}}}{\mathrm{minimize}}}\quad l(\tilde{x},\mu(\tilde{x}|\theta^{\mu}))+V_{\theta^{V}}(f(\tilde{x},\mu(\tilde{x}|\theta^{\mu}))) (2a) where \mu(\cdot) and \theta^{\mu} are the actor network and its parameters. The improved actor policy then generates rollouts that warm-start the subsequent TO problems, closing the loop between TO and RL. In [ 15 ] , CACTO was shown to outperform other RL algorithms such as DDPG [ 25 ] and PPO [ 35 ] as warm-start provider in terms of training time. III BIASED INITIAL STATES SAMPLING Identifying regions of the state space with high potential for policy improvement is a challenging problem. Exploration strategies in RL can be broadly categorized into undirected and directed approaches. Undirected exploration relies on stochastic action selection, such as \epsilon -greedy policies, to explore the state space without explicitly accounting for uncertainty or novelty. Directed exploration, in contrast, uses signals derived from the agent’s learning process or auxiliary models to guide behavior toward less familiar or more uncertain regions. A common class of directed methods employs intrinsic rewards or exploration bonuses, including count-based or approximate count methods [ 4 ] , uncertainty metrics [ 26 ] , and curiosity-driven approaches such as prediction-error bonuses and random network distillation [ 30 , 7 ] . Several works have also explored initial-state or restart-based exploration. These approaches often assign scores to states to prioritize which ones to explore further. Criteria for scoring include the system’s sensitivity [ 29 ] , the familiarity [ 34 ] , the uncertainty [ 43 ] , or the TD error [ 38 ] . [ 27 ] , proposes a structured replay buffer that groups states by task relevance and prioritizes sampling from unmastered sub-tasks. When available, prior or expert knowledge can also be leveraged to further guide exploration. We address the exploration problem by leveraging the insight that the value function \bar{V}(x) associated with locally optimal solutions is generally piecewise continuous . III-A Motivating Example: Discontinuous Value Function Let us illustrate how the structure of the value function associated with a locally optimal policy can help us tackle the exploration problem. We focus on a toy problem with a 1D state, single integrator dynamics, and a cost with two local minima (see Fig. 1 ). Solving TO problems with a naive initial guess, highlights the presence of two basins of attraction, \mathcal{R}_{1} , and \mathcal{R}_{2} , each corresponding to a different local minimum (see Fig. 1 ). Fig. 1: Cost and Value obtained with TO using a naive initial guess. The critic smooths the Value’s discontinuities. Within each basin, the real Value \bar{V} is continuous, but it is discontinuous at the shared boundary \mathcal{R}_{1}\cap\mathcal{R}_{2}\triangleq\partial\mathcal{R} . In the neighborhood of \partial\mathcal{R} , \bar{V} is lower (i.e. better) in \mathcal{R}_{1} than in \mathcal{R}_{2} . After training the critic with the first batch of TO episodes, the network smooths out the discontinuity (see Fig. 1 ) in a region \mathcal{N} around \partial\mathcal{R} , where the critic either underestimates or overestimates the true value \bar{V} : \displaystyle V(\bar{x}|\theta_{V})>\bar{V}(\bar{x})\qquad\forall\bar{x}\in\mathcal{N}\cap\mathcal{R}_{1} (3) \displaystyle V(\bar{x}|\theta_{V})<\bar{V}(\bar{x})\qquad\forall\bar{x}\in\mathcal{N}\cap\mathcal{R}_{2} (4) Therefore in \mathcal{N} the critic’s gradient will point towards \mathcal{R}_{2} . Due to this gradient, during the policy improvement phase the actor can improve in \mathcal{N}\cap\mathcal{R}_{2} , learning to steer the state toward \mathcal{R}_{1} rather than \mathcal{R}_{2} . This example reveals that the regions near value function’s discontinuities hold great potential for policy improvement. III-B General case: Discontinuous Value Function The phenomenon shown in Section III-A frequently occurs in problems with multiple local minima. Each local minimum defines a basin of attraction where the value function is continuous, while discontinuities typically appear at the boundaries between basins. These discontinuities highlight regions with great potential for policy improvement. In contrast, sampling initial states far from such discontinuities likely leads the actor to simply imitate TO. III-C Detecting Informative Regions via Critic Uncertainty Approximating the value function close to the discontinuities is particularly challenging for the critic network, which cannot accurately represent abrupt changes due to its continuous activation functions. As a result, the critic tends to incur larger errors near these boundaries. We suggest leveraging the uncertainty estimation of the critic’s output to identify the value’s discontinuities. An additional neural network, referred to as std-critic , is introduced to predict the standard deviation of the critic. Following [ 37 ] , this network is trained at the end of each actor–critic update phase by minimizing the negative log-likelihood of a normal distribution: \displaystyle\mathmakebox[\widthof{$\underset{\displaystyle\scriptstyle{\theta^{\text{std}}}}{\mathrm{subject~to}}$}][l]{\underset{\displaystyle\scriptstyle{\theta^{\text{std}}}}{\mathrm{minimize}}}\quad\ln(V^{\text{std}}(\tilde{x}|\theta^{\text{std}}))+\frac{1}{2}\frac{(\bar{V}-V(\tilde{x}|\theta^{V}))^{2}}{V^{\text{std}}(\tilde{x}|\theta^{\text{std}})^{2}} (5a) where V^{\text{std}}(\cdot) and \theta^{\text{std}} are the std-critic network and its parameters. The first term of this loss pushes V^{std} to be small everywhere, while the second term pushes V^{\text{std}} to increase when the critic’s error is large. We train this network after the critic to avoid issues during the early stages of learning. The initial states with the highest potential for policy improvement are then selected according to their predicted uncertainty V^{\text{std}}(\tilde{x}) . III-D Algorithm Overview An overview of the CACTO-BIC (Biased Initial Conditions) algorithm is illustrated in Fig. 2 . Fig. 2: Overview of CACTO-BIC with biased initial-state sampling. During the first iteration, as in the original CACTO framework [ 15 ] , initial states for the TO problems are sampled uniformly at random. From the second iteration onward, a set of 10N candidate initial states is sampled and ranked based on V^{\text{std}} . The top N samples are then selected for the next TO batch. Moreover, since trajectories starting from these initial states provide more information, we can reduce the number of problems solved from the second iteration (see Section V-A for details). IV GPU-based computation To reduce the computation time of the algorithm by leveraging GPUs, the entire framework was migrated to JAX, a Python library for accelerator-oriented array computation and program transformation [ 6 ] . The migration is beneficial in two ways: first, migrating to GPU the neural network training—accounting for about 90% of CACTO’s total computation time— yields significant speedups; second, performing TO directly on GPU eliminates any CPU-GPU data-transfer overhead, in addition to reducing TO’s computation time, which significantly grows with the complexity of the system. Migrating to GPU-based computation required some adaptations to meet the constraints of parallel processing: GPUs require fixed-size arrays and uniform computation across batches to enable efficient vectorization, leading to the following two key challenges. IV-A Fixed number of iterations Because GPUs execute batched computations in parallel, all TO problems in a batch must undergo the same number of optimization iterations. Consequently, the maximum number of iterations, max\_iter , becomes a crucial hyperparameter. Setting this value too low may lead to premature termination, and hence to collect too few or insufficiently informative samples. In contrast, setting it too large can cause longer runtimes, since hard-to-converge problems dominate the batch’s overall computation time, slowing the entire pipeline. Moreover, the ideal number of iterations may vary as the policy improves, as better initial guesses generally lead to faster convergence. To determine the maximum number of iterations, we suggest to solve a large set of problems using a naive warm-start and a high iteration limit ( max\_iter=1000 ), generating a dataset of the iterations needed to converge. The maximum number of iterations can then be set to the 99th percentile of iteration counts for the first iteration of CACTO (where the naive warm-start is used) and to the 50th percentile for subsequent iterations (where CACTO’s actor provides the warm-start). IV-B Regularization for matrix inversion In iLQR [ 18 ] , it is crucial to regularize the Hessian of the value function so that it is positive definite. The standard regularizationconsists in increasing the local control-cost Hessian with a diagonal term: \tilde{Q}_{uu}=Q+\mu I_{m}, where \mu plays the role of a Levenberg-Marquardt parameter. During the backward pass, if the line search fails, \mu is increased and the backward pass is retried, otherwise, \mu is decreased. This approach is computationally efficient when solving problems sequentially on a CPU. However, it becomes impractical when solving multiple problems in parallel on a GPU, where such iterative tuning can significantly slow down the batched computation. This issue is addressed through regularization based on the eigenvalue decomposition. The eigenvalues S and corresponding eigenvectors W of the state-cost Hessian and the control-cost Hessian are computed, the eigenvalues are clipped from below by a user-defined constant \epsilon>0 , S^{\prime}=\max(S,\epsilon) , and the matrices are reconstructed: \displaystyle Q_{+} \displaystyle=W\,\text{diag}(S^{\prime})\,W^{T},\quad Q_{psd} \displaystyle=\frac{1}{2}(Q_{+}+Q_{+}^{T}) (6) This procedure handles poorly-conditioned Hessians, while avoiding additional iterative steps. As a result, it is better suited for large-batch processing on GPUs. IV-C Implementation Details The system dynamics and cost functions, implemented using the CasADi library [ 3 ] in CACTO-SL [ 1 ] , were converted into JAX-compatible functions through the Jaxadi library [ 2 ] . For TO, we replaced the previous solver with an iLQR implementation from the Trajax library [ 13 ] , which provides the additional benefit of computing the gradient of the value function (used for training the critic network) while solving the optimal control problem. The neural networks were implemented using the Flax library [ 17 ] , allowing the entire pipeline to remain on the GPU. Our new implementation is open-source and available on the GitHub page of the project . V RESULTS This section presents our evaluation of CACTO-BIC. First, we assess the impact of the biased initial-state sampling on data efficiency (Section V-A ). Second, we analyze the computational benefits of the GPU-based implementation (Section V-B ). Third, we compare CACTO-BIC with a state-of-the-art RL algorithm (Section V-C ). Finally, we demonstrate the scalability of the approach through experiments on a high-dimensional quadruped robot, AlienGO [ 39 ] (Section V-D ). V-A Biased Initial State Sampling We evaluate the proposed exploration method on the same benchmark scenarios used in [ 15 , 1 ] . The task consists in minimizing the distance between the robot’s end-effector and a target, while avoiding three elliptical obstacles (encoded with large penalties) and minimizing control effort. An additional reward is provided in the neighborhood of the target, as shown in Fig. 3 . Fig. 3: Cost function excluding the control effort term, with target set at [-7,0] . When the system starts from the Hard Region (highlighted in Fig. 3 ), it becomes challenging for a gradient-based solver to converge to the globally optimal solution. We consider three systems of increasing complexity: a point mass with state (x,y,v_{x},v_{y},t)\in\mathbb{R}^{5} and control (a_{x},a_{y})\in\mathbb{R}^{2} , a jerk-controlled version of the Dubins car model [ 11 ] with state (x,y,\theta,v,a,t)\in\mathbb{R}^{6} and control (\omega,j)\in\mathbb{R}^{2} , and a 3-degree-of-freedom (DoF) planar manipulator with a 7-dimensional state space and a 3-dimensional control input. We compare three algorithms: 1. CACTO as presented in [ 1 ] ; 2. CACTO-BIC: CACTO with biased initial-state sampling and a reduced number of TO episodes (25%) from the 2nd iteration onward; 3. CACTO with reduced TO episodes (as CACTO-BIC), but without biased initial-state sampling. All algorithms run for the same number of learning iterations (i.e. updates of the neural networks). Fig. 4 shows the median (across 5 runs) of the mean cost (across initial conditions) as a function of the number of TO episodes. The results show that CACTO-BIC achieves comparable performance using 30-40\% of the number of TO episodes. However, this improvement comes with increased training time due to the additional std-critic network. With CACTO-BIC, the percentage of computation time devoted to training the networks rises to 93-94\% . This motivates moving to the GPU. 0 0.2 0.4 0.6 0.8 1 1.2 1.4 \cdot 10^{3} 0 50 Mean cost CACTO CACTO w. BIC (0.25 TO) CACTO (0.25 TO) 0 1 2 3 4 5 6 7 8 9 \cdot 10^{3} -10 0 10 Mean cost 0 0.2 0.4 0.6 0.8 1 1.2 1.4 \cdot 10^{4} 0 20 40 # TO episodes Mean cost Fig. 4: Median (across 5 runs) of the mean cost (across initial conditions) starting from the Hard Region for the point mass (top), the Dubins car , and the manipulator (bottom). Shaded areas represent first and third quartiles. Data are sampled every n updates. When multiple measurements occur at the same TO episode count, the last one is plotted. V-B Computational Efficiency of GPU Computation First, we assessed the speedup achievable when solving a batch of TO problems on a GPU for different batch sizes: [10,50,100,250,500,1000,5000,10000] . Fig. 5 shows that while the CPU computation time scales approximately linearly with batch size, the GPU benefits from parallelization: the time per problem decreases as the batch size increases, eventually saturating, demonstrating the advantage of GPU-based TO. 1 1.5 2 2.5 3 3.5 4 0 10 20 Log TO batch size Speedup: \displaystyle\frac{t_{\mathrm{CPU}}}{t_{\mathrm{GPU}}} Point Mass Dubin Car Manipulator Speedup = 1 Fig. 5: Speedups for TO on GPU (w.r.t. CPU) for the point mass, the Dubins car , and the manipulator. Dashed lines represent speedups with random initial states and maximum number of iterations, selected as described in Section IV-A , while solid lines represent speedups with initial conditions selected using CACTO-BIC and maximum number of iterations set as in the two versions. We now analyze the speedup achieved through the new GPU-accelerated implementation of CACTO-BIC. The number of TO problems solved in parallel was 300, 500, and 550 in the first iteration for the point mass, Dubins car and manipulator, respectively. From the second iteration onward, the batch size was reduced to 25%. The initial time is set to 0 in each TO instance. Table I reports the computation times to perform the same updates using three versions of CACTO-BIC: a single-thread CPU version, a multi-thread CPU version, and the novel GPU version. TABLE I: Comparison between CPU and GPU versions of CACTO-BIC. GPU runtimes do not include the warm-up phase required for the JIT compilation of the TO solver. System (nb. of updates) CPU GPU 1 core 10 cores Point Mass (30k) 23 min 22 min 44 s Dubins Car (200k) 3 h 44 min 3 h 34 min 7 min Manipulator (330k) 5 h 35 min 5 h 15 min 6 min This test was performed on a workstation equipped with an AMD Ryzen 9 7950X CPU, 192 GB of RAM, and an NVIDIA RTX 6000 GPU with 48 GB VRAM, running on Ubuntu 22.04. The results show that using 10 cores to parallelize the TO problems has little effect on the total computation time as most time is spent for training the neural networks. The novel GPU version of CACTO-BIC achieves instead remarkable speedups ( \approx 30x for the point mass and Dubins car, and \approx 56x for the manipulator). Although GPUs have far more cores than CPUs, the resulting speedups are smaller than this hardware difference suggests. Several factors contribute to this limitation. First, the TO problems are solved in batches on the GPU. Therefore a few hard instances may dominate the total computation time and cap the achievable speedup. Second, computations rely on reduced numerical precision. Although this improves raw throughput, it can introduce numerical instability and increase the number of required iterations. By analyzing the time spent in the TO phase (creating the warm-start and solving TO problems) and in the RL phase (training networks), we observe that their relative contribution varies across systems, but remains comparatively balanced. Specifically, the TO and RL phases account for approximately 42% and 53% of the total time in the point mass, 61% and 37% in the Dubins car, and 22% and 75% in the manipulator. It follows that the speedup comes mainly from training the networks on the GPU, which is 51-85 \times faster than on the CPU in our tests.In contrast, the speedup in the TO phase is modest (3-5 \times ) for the Dubins car and point mass, but it becomes significant (19 \times ) for the most complex system. V-C Comparison with PPO To evaluate CACTO-BIC w.r.t. state-of-the-art RL, we selected Proximal Policy Optimization (PPO) [ 35 ] as a benchmark. For a fair comparison, we employed the fully GPU-based implementation of PPO provided by BRAX [ 12 ] . Our analysis focuses on two aspects: first, learning a warm-start policy for a problem that exhibits local minima; second, learning a control policy for a classic benchmark problem, a customized version of the Reacher environment. Fig. 6 reports the mean cost obtained using CACTO-BIC’s and PPO’s policies as warm-start for TO in the Point Mass and Manipulator environments. CACTO-BIC converged in one-third of the time in the first environment and in just 7% in the second one. 0 10 20 30 40 50 60 70 80 90 100 -40 -20 0 Mean cost CACTO-BIC PPO (Brax) 0 0.5 1 1.5 2 2.5 \cdot 10^{4} 0 10 20 30 Computation time [s] Mean cost CACTO-BIC PPO (Brax) Fig. 6: Warm-start provider comparison: Median (across 5 runs) of the mean cost (across initial conditions) starting from the Hard Region for the point mass (top), and the manipulator (bottom). Shaded areas represent first and third quartiles. In Fig. 7 we compare CACTO-BIC and PPO in the Reacher Environment. Also in this case, CACTO-BIC achieves a similar cost in \approx 10% of the computation time. 0 50 100 150 200 250 300 350 400 4 5 6 7 Computation time [s] Mean cost CACTO-BIC PPO (Brax) Fig. 7: Policy comparison: Median (across 5 runs) of the mean cost (across initial conditions) for a customized Reacher environment. Shaded areas represent first and third quartiles. V-D Hardware experiments To evaluate CACTO’s scalability and its potential for real-time applications we tested it on AlienGO, a quadrupedal robot featuring 12 degrees of freedom [ 39 ] . We address the problem of navigation in confined environments, with even terrain, and the presence of a moving obstacle. The robot must reach a moving target (see Fig. 8 ) while avoiding collisions with both the moving obstacle (a sphere of radius 0.5 m) and the walls of the room (a rectangle with sides ranging from 2 m to 10 m). We consider a fixed trotting gait with alternating diagonal leg pairs making contact with the ground. Since legged robots have non-differentiable dynamics [ 41 ] , which are not compatible with gradient-based TO, we adopt a hierarchical approach. We combine a high-level policy relying on a simplified differentiable model, with a low-level policy using the complete robot dynamics. The high-level policy is trained with CACTO-BIC, while the low-level policy is trained with the RL algorithm CAT [ 10 ] , which can handle non-differentiable dynamics. In practice, CACTO-BIC’s actor is used as a standalone policy to provide reference trajectories to the low-level policy. V-D 1 High-level policy To train this policy, we employed a nonlinear version of the Linear Inverted Pendulum Model [ 21 ] . The state x\in\mathbb{R}^{15} comprises the 2D offsets of the front and rear support feet relative to the associated shoulders \Delta p_{f} and \Delta p_{r} , the Center of Mass (CoM) position c\in\mathbb{R}^{2} and velocity \dot{c}\in\mathbb{R}^{2} on the horizontal plane, the step index s_{idx}\in\mathbb{N} , which encodes both the current contact phase and time, the obstacle position c_{obs}\in\mathbb{R}^{2} and the location of the four walls, expressed as offsets with respect to the global reference frame \Delta_{walls}\in\mathbb{R}^{4} . x\triangleq(\Delta p_{f},\Delta p_{r},c,\dot{c},s_{idx},c_{obs},\Delta_{walls})\in\mathbb{R}^{15}. (7) The target position is assumed to be the origin. Assuming a trotting gait and constant Center of Pressure (CoP) during each contact phase, the control vector u\in\mathbb{R}^{6} includes the offsets of front and rear support feet w.r.t. the associated shoulder in the next contact phase, \Delta p_{f}^{+},\Delta p_{r}^{+}\in\mathbb{R}^{2} , a scalar \alpha\in[0,1] that expresses the CoP as a convex combination of the support foot positions, and the contact phase duration \delta t : u\triangleq(\Delta p_{f}^{+},\Delta p_{r}^{+},\alpha,\delta t)\in\mathbb{R}^{6} (8) The cost function penalizes the CoM-target distance, the CoM velocities, and a barrier-like cost penalizes CoM velocities beyond prescribed bounds. A smooth logarithmic reward is used to encourage reaching a narrow region around the target. Control regularization discourages deviations from nominal values: zero foot displacements, centered CoP ( \alpha=0.5 ), and nominal contact-phase duration ( \delta t=0.375 s). Obstacle avoidance is enforced through smooth logarithmic penalties. The algorithm took \approx 109 s to converge. In this scenario, leveraging the GPU for solving the TO problems yields a substantial speedup, roughly 345 \times . V-D 2 Low-level policy The low-level policy is trained using a constrained RL algorithm [ 10 ] , which builds upon PPO [ 35 ] , employing Isaac Gym for GPU simulation. The goal of this policy is to track the references generated by the high-level policy. During training, the high-level policy is rolled out in open loop, and updated only at the beginning of each new contact phase to enhance stability and robustness [ 40 ] . The low-level policy receives as observations: the base state (position and orientation, linear and angular velocity, and projected gravity), the target base position and linear velocity, yaw orientation error, foot placement errors, a binary flag indicating the active diagonal contact pair, the remaining time in the current gait phase, joint positions, velocities and the previous action. The reward function encourages accurate tracking of base position and orientation, linear and angular velocity, and the desired foot contact locations. Regularization terms are included to improve smoothness. All safety and style-related constraints are divided into soft constraints (which define a termination probability as a function of the constraint violation) and hard constraints (which cause immediate termination of the episode). V-D 3 Aliengo Simulation and Hardware results The experiments are conducted in an indoor area of 4 m \times 4 m (see Fig. 8 ). The obstacle and the target are either stationary or manually actuated, depending on the experiment. Fig. 8: Experimental setup: The lines define a 4 m \times 4 m operational area. State feedback is obtained from a motion capture system, which measures position and orientation of the robot, the obstacle and the target. Velocities are estimated by finite differencing and low-pass filtering. Results are shown in the accompanying video. VI CONCLUSIONS This paper presented CACTO-BIC, an extension of CACTO designed to improve scalability and computational efficiency in combined TO-RL framework by biasing the initial-state sampling using value function properties and leveraging GPU acceleration. Experimental results demonstrate that the proposed sampling strategy effectively identifies state-space regions where actor policy improvement is more likely, leading to a 2.5-3.5\times increase in sample efficiency. In addition, GPU-based computation achieves 30-250 \times speedup compared to CACTO, with increasing benefits as system complexity grows. When compared to PPO, CACTO-BIC achieves similar final costs requiring only 7-30\% of the training time. Finally, experiments on the AlienGO quadruped robot show that CACTO-BIC scales to high-dimensional robotic systems and can be effectively used for real robot control. Future works will focus on extending the proposed approach to handle constraints using augmented Lagrangian formulations [ 23 ] . In addition, we will explore the integration of sampling-based optimization techniques, such as MPPI [ 42 ] , to tackle non-differentiable dynamics. Finally, applying domain randomization may improve robustness to uncertain dynamics enhancing generalization to real-world systems [ 31 ] .