CRoSS: A Continual Robotic Simulation Suite for Scalable Reinforcement Learning with High Task Diversity and Realistic Physics Simulation Yannick Denker, Alexander Gepperth Abstract Continual reinforcement learning (CRL) requires agents to learn from a sequence of tasks without forgetting previously acquired policies. In this work, we introduce a novel benchmark suite for CRL based on realistically simulated robots in the Gazebo simulator. Our Continual Robotic Simulation Suite (CRoSS) benchmarks rely on two robotic platforms: a two-wheeled differential-drive robot with lidar, camera and bumper sensor, and a robotic arm with seven joints. The former represent an agent in line-following and object-pushing scenarios, where variation of visual and structural parameters yields a large number of distinct tasks, whereas the latter is used in two goal-reaching scenarios with high-level cartesian hand position control (modeled after the Continual World benchmark), and low-level control based on joint angles. For the robotic arm benchmarks, we provide additional kinematics-only variants that bypass the need for physical simulation (as long as no sensor readings are required), and which can be run two orders of magnitude faster. CRoSS is designed to be easily extensible and enables controlled studies of continual reinforcement learning in robotic settings with high physical realism, and in particular allow the use of almost arbitrary simulated sensors. To ensure reproducibility and ease of use, we provide a containerized setup (Apptainer) that runs out-of-the-box, and report performances of standard RL algorithms, including Deep Q-Networks (DQN) and policy gradient methods. This highlights the suitability as a scalable and reproducible benchmark for CRL research. 1 Introduction Continual reinforcement learning (CRL) extends the classical reinforcement learning (RL) framework to settings where an agent must learn a sequence of tasks over time without access to previous environments. In contrast to standard RL, where the environment is typically assumed to be stationary and training data can be revisited indefinitely, CRL requires agents to continuously integrate new knowledge while preserving performance on previously encountered tasks. This continual setting is relevant for real-world applications such as robotics, autonomous driving, or online decision-making systems, where environmental conditions change over time and retraining from scratch is infeasible. A core difficulty in CRL is catastrophic forgetting 21 ; 28 ; 23 ; 18 , where newly acquired knowledge interferes with previously learned policies. Although this problem is similar to to the one faced in Continual Learning (CL), which deals with learning from non-stationary distributions, but without environment interaction, many established CL methods cannot easily be generalized to CRL. In particular, the assumption that each task is composed of well-defined sample classes , and that task onsets and classes are known to the learner, is what distinguishes CL from CRL. 1.1 Motivation Current CRL research commonly relies on benchmarks such as robotic reaching suites (e.g., Continual World 34 ) and sequential game-based environments (e.g., Atari-based continual learning setups 1 ; 5 ). While these benchmarks have been valuable for early progress, they exhibit several limitations that constrain their applicability. Robotic CRL benchmarks typically offer only a small number of distinct tasks, with limited support for sensors, plugins or actuator extensions. Many of these environments also do not work ”out of the box”, requiring substantial installation effort and relying on simulation backends that are difficult to configure, distribute, or extend. For Continual World, observations are low-dimensional, and the tasks do not really require learning in the first place, since the robot arm could be controlled directly using inverse kinematics. Additionally, the simulation is kinematics-only, containing no real physics engine. Game-based continual RL setups, including Atari, provide high-dimensional observations and actions with a wide variety of tasks. However, they operate in noise-free, fully deterministic environments that lack physical realism. Tasks are very diverse, but their intrinsic difficulty is very high, making it difficult to separate intrinsic difficulty from the difficulty of continual learning. We thus require a benchmark suite that incorporates out-of-the-box readiness, physical realism, a well-supported and extensible simulation backend, and a large number of tasks with high diversity. Our goal is to close this gap by introducing the Continual Robotic Simulation Suite ( CRoSS ), a Gazebo-based CRL benchmark suite that combines realistic robotic simulation with massive task diversity while providing a reproducible, containerized deployment. CRoSS is designed around two complementary robotic platforms: First of all, a two-wheeled differential-drive robot is used in multi-task line-following (MLF) and multi-task pushing-objects (MPO) scenarios, where visual and structural parameters (e.g., line shape, textures, colors) are systematically varied to produce hundreds of distinct (but relatively simple) tasks. On the other hand, a 7-d.o.f. robotic arm is evaluated in high-level reaching (Cartesian end-effector control in six discrete directions mirroring the setup of Continual World) and low-level reaching , where the agent directly manipulates the seven arm joints, increasing task dimensionality and realism. Our benchmark design leverages the Gazebo-Transport middleware communication between sensors, actuators, and agents. Since Gazebo and the Robot Operating System (ROS) are compatible through a standardized bridge, this control interface can be directly applied to physical robots with minimal modification, supporting extensions toward sim-to-real continual learning. Our environments can be used as drop-in replacements for standard Gymnasium tasks. The environment manager follows the same API conventions as Gymnasium making the benchmark directly compatible with existing RL pipelines and libraries. Through containerization, installation is simplified across a wide range of platforms. 1.2 Contribution We introduce a new benchmark suite for CRL with the following key contributions: Realistic robotic environments: We present two Gazebo-based robotic platforms, a 2-wheeled differential-drive robot and a 7-d.o.f. robot arm, enabling embodied CRL experiments in physically consistent simulations. High task diversity and scalability: By varying visual and structural parameters, our benchmark generates hundreds of distinct tasks, allowing controlled studies of forgetting, transfer, and scalability in continual learning. Multiple control modalities: The robotic arm is controlled both in cartesian and joint-space mode, allowing varying action complexities. Standardized, reproducible and extensible setup: All environments are provided in a containerized (Apptainer) format that runs out of the box on any Linux system, using a well-supported simulation back-end that can be extended by a variety of community-developed plugins. Baseline evaluations and metrics: We benchmark standard CRL algorithms, including DQN and REINFORCE, and provide evaluation metrics for measuring forgetting, transfer, and final performance. Simulation-to-real compatibility: The benchmark’s communication framework is built on Gazebo-Transport, which is inter-operable with ROS via an existing bridge, making it straightforward to port trained agents or environments from simulation to hardware platforms. Together, these contributions establish a scalable, realistic, and easily extensible foundation for advancing continual reinforcement learning research in robotic domains. All benchmark environments, experiment scripts, and documentation are available in our public repository 1 1 1 https://github.com/anon-scientist/continual-robotic-simulation-suite/ . 2 Related Work Comprehensive surveys have outlined the current state and challenges of the CRL field 12 ; 20 ; 8 . These works summarize existing methods and highlight the open problems in scalability, stability, and evaluation that motivate continued research in realistic and dynamic environments. More recently, generative replay and task-agnostic methods have emerged that adapt without explicit task boundaries (e.g., 17 ; 11 ; 35 ) While these advances have improved algorithmic robustness, evaluation often remains confined to synthetic or low-dimensional environments that do not reflect the challenges of embodied learning. Benchmark design has played a critical role in the development of CL algorithms. Classical benchmarks such as Split-MNIST, Permuted-MNIST, or CIFAR-100 task sequences 6 ; 36 ; 14 ; 39 ; 33 primarily target supervised continual learning. In CRL, early environments such as Atari-100k or ProcGen 3 ; 2 provided diverse visual tasks, though they lack the continuous control and physical realism required for robotic studies. The Continual World benchmark 34 extended the well-known Meta-World suite 38 to a continual setting, defining sequences of manipulation tasks controlled through simple 3D end-effector movements. A recent survey of continual RL benchmarks 25 provides a broad overview of existing CRL benchmarks. These can be roughly grouped into game-based environments 5 ; 27 ; 32 ; 16 ; 13 ; 29 ; 10 ; 22 ; 19 which offer visual diversity but lack realism and robotic or physics-based benchmarks 34 ; 37 ; 26 ; 9 ; 30 , which provide physical interaction but typically only few tasks, simplified observations such as perfect 3D positions or lack mechanisms for scalable task generation. Frameworks such as MuJoCo, PyBullet, and Gazebo 31 ; 4 ; 15 ; 7 have enabled scalable robotic experiments, though most have been used for isolated single-task RL. 3 Simulated robots 3.1 7-d.o.f. robot arm This scenario employs a simulated, seven-degree-of-freedom (7-d.o.f.) robotic arm modeled after the Franka Emika Panda, see Fig. 1 for a visual impression. The arm consists of seven joints providing a flexible kinematic structure that enables dexterous positioning of the end-effector in three-dimensional space. Each joint is actuated independently and reports its angular position and velocity, allowing precise low-level control. Ground-truth information, such as the end-effector pose, can be retrieved from the simulator for reward generation and policy evaluation. 3.2 Two-wheeled mobile robot Figure 1 : Simulated robots. Left: two-wheeled 3\pi robot. Lidar sensors and camera objects are just visual placeholders to indicate their positions. Right: 7-d.o.f. Franka Emika Panda arm. This is a two-wheeled, differential-drive robot modeled after the Pololu 3Pi, see Fig. 1 for a visual impression. The wheel separation is 10cm, and the wheel radii are 0.5cm each. It is equipped with a contact (bumper) sensor of radius 1cm placed in front, a lidar sensor placed in the center of the robot at a height over the ground of 4cm, and an RGB camera in various configurations. Although the position and orientation can be obtained from the simulator, we only use sensor data that could be obtained from a real robot as well. The lidar sensor covers a forward-looking angle range of 4\pi using 500 beams at a frequency of 300 Hz. Camera sensors can be configured to look down or ahead with a variable resolution. In addition to the differential-drive effector, the robot is equipped with 6 LEDs (red, blue, green, magenta, yellow, white) that can be controlled individually. 4 Benchmarks 4.1 Multi-task line following (MLF) For this benchmark, the two-wheeled robot ( Section 3.2 ) is equipped with a line camera that is downwards-oriented for observing the floor ahead of the robot, and has a resolution of 50x3 24-bit color pixels. The environment is designed to produce a large number (150) of visually distinct tasks through variations in line and ground colors, as well as optional behavior-based modifiers that increase task diversity. The robot moves on a terrain covered with tracks of length 0.75m, consisting of 3 colored lines each (with the center line being slimmer). At the end of each track, there is a wall obstacle. The line-following tasks are generated from six available colors, resulting in 150 unique tracks. Tracks in which two adjacent lines would have the same color are excluded. The robot is rewarded for following the left side of the central line, and it equally receives rewards/punishments for turning on the correct lamp at the correct time: in the first half of the track, the robot must turn on a certain lamp determined by the colors of the three lines, and another for the second half of the track. The robot can measure its distance from the end of the track using its lidar sensor. One task in this benchmark consists of 50 episodes of 30 steps at most, randomly placing the robot at the start of one out of four different tracks (which tracks are used is a property of the task) and letting the robot try to follow the inner line while turning on the correct lamps. From one task to the next, one track is replaced by another that has not been seen before. The action space is 18-dimensional: the three basic actions ”right curve”, ”left curve” and ”straight” are duplicated 6 times for each of the lamps that should be turned on. There is no stopping action. The wheel speeds for all actions are tabulated in Table 6 . They are chosen such that forward speed is always constant, and that this constant forward speed is not sufficient to reach the wall in a single episode. The observation space is represented by a 100x3 RGB image from the robot’s line camera. The distance to the end of the track, as measured by the lidar sensor, is embedded into the third row of the visual image. For details of reward computation, see Section C.1 . This benchmark can be configured in two additional ways besides the default setting (DS) described above, leading to progressively easier variants of the benchmark. The goal is to make individual tasks easier, in order to focus more on the difficulty of continual learning with many tasks rather than on the difficulty of learning one particular task. In the simplified setting (SS), the visual signal is transformed to an observation space of 3x15 mono images encoding the behavior-relevant aspects of wall distance d_{t} , transition pixel position t_{t} , left-line color L_{t} , middle-line color M_{t} and right-line color R_{t} in an unambiguous way, see Fig. 3 for details. The information for achieving this is taken from the lidar sensor for d_{t} , from the known properties L_{t} , M_{t} and R_{t} of the track the robot is currently placed on, and from image processing of the current observation \vec{o}_{t} for the transition pixel position t_{t} . In the super-simplified setting (SSS), the visual signal is transformed as in the SS. In addition, the action space of the DNN controller is reduced to the six actions ”LED1” through ”LED6”. The direction of motion (”straight”, ”left” or ”right”, see Table 6 ) is determined by a non-adaptive controller based on the known value of t_{t} . Please see Table 7 for an overview of observation and action spaces in all three settings for this benchmark. 4.2 Multi-task pushing objects (MPO) Figure 2 : Benchmarks for the two-wheeled robot used in this study. Left: multi-task pushing objects (MPO), the robot is approaching geometrical objects with different shapes, colors, and symbols projected onto their faces. Right: multi-task line following (MLF), the robot drives on a ground plane covered with colored lines and limited by a wall. It is supposed to follow the slim centered lines. The robot is the same in both benchmarks but uses a downwards-looking line camera in the MLF benchmark, whereas a forward-looking RGB camera is used in the MPO benchmark. Figure 3 : Sensor data used for learning in the MLF and MPO benchmarks. Left: two default-setting 20x20 RGB images and two simplified 15x3 mono images from the MPO benchmark. Simplified inputs are population-coded, meaning that the position of the single bright pixel indicates a sensor reading. The uppermost row is split into three population codes for color, shape and symbol of the currently approached object, whereas the next rows encode object distance and viewing angle. Right: Two default-setting 100x2 composite images and two simplified 18x3 images from the MLF benchmark. Composite images integrate line camera data (upper row) and lidar distance to wall (lower row), the latter again population-coded. Simplified images contain three population codes for left/middle/right line color (upper row), distance to wall (middle row) and centering of the left border of the middle line in the image (population-coded). The simplified inputs are helpful because they simplify the problems without sacrificing their massively continual character. In this benchmark, the robot of Section 3.2 is placed before one of 150 possible objects. It is always rewarded for approaching them, and punished or rewarded for driving into them (”pushing”). Whether an object incurs a reward (”pushable object”) or punishment (”non-pushable object”) if the robot drives into it, depends on the shape, color and symbol place on the faces of the object. One approach episode of at most 30 steps, with a target object of distinct shape, color and symbol, is termed a track . Since there are 5 choices for color (white, pink, red, green, blue), 6 choices for symbols ( \# , A, X, O, plain, H) and 5 choices for shape(sphere, cube, pyramid, diamond, cylinder), there is a total of 150 possible tracks . One task consists of four tracks depending on the task, and only one track is replaced by an unseen one from one task to the next, leading to significant task overlap. Each episode begins with the robot positioned at a distance of .45m in front of an object, with a randomized orientation of \pm 18^{\circ} relative to the object’s direction. This angular offset prevents the robot from reaching the object by chance and requires directed action. There are four possible actions a_{t} : ”turn left”, ”turn right”, ”straight” and ”stop”, see Table 6 . Observations \vec{o}_{t} are 20x20 RGB images of the robot’s front view camera, see Fig. 3 for same examples. For details of reward computation, see Section C.2 . This benchmark can be configured in two additional ways besides the default setting (DS) described above, leading to progressively easier variants of the benchmark. The goal is to make individual tasks easier, in order to focus more on the difficulty of continual learning with many tasks. In the simplified setting (SS), the visual signal is transformed to an observation space of 3x15 mono images encoding the behavior-relevant aspects of object distance d_{t} , viewing angle \phi_{t} , color c_{t} , form f_{t} and depicted symbol s_{t} in an unambiguous way, see Fig. 3 for details. The information for achieving this is taken from the lidar sensor for \phi_{t} and d_{t} , as well as the known properties c_{t} , f_{t} and s_{t} of the track the robot is currently placed on. In the super-simplified setting (SSS), the visual signal is transformed as in the SS. In addition, the action space of the DNN controller is reduced to the two actions ”stop” and ”go”, where the latter indicates that the full action (”straight”, ”turn left” or ”turn right”, see Table 6 ) should be determined by a non-adaptive controller based on the known value of \phi_{t} . Please see Table 7 for an overview of observation and action spaces in all three settings. 4.3 High-Level Reaching (HLR) This benchmark mirrors the simplified setup used in Continual World and involves reaching target positions with the end-effector (”hand”) of the simulated Franka Emika Panda arm from Section 3.1 . Tasks are characterized by the 3D target positions, and episodes always last for 30 steps. The agent controls the robot arm in Cartesian space using 6 discrete actions that move the hand along the three axes ( \pm x , \pm y , \pm z ). The observation includes the current and goal 3D positions of the end-effector. Further details about observation and actions spaces are provided in Section B.5 . Upon choosing an action, the new target position is passed to PyKDL 2 2 2 https://docs.ros.org/en/diamondback/api/kdl/html/python/ which computes the corresponding joint angles by inverse kinematics. These angles are directly applied to the arm using the Gazebo IKPanda plugin. If the desired position violates joint limits or is otherwise unreachable, the movement is aborted and a penalty is applied. For details of reward computation, see Section C.4 . 4.4 Kinematic High-Level Reaching (HLR-K) In addition to the physics-based simulation, we provide a kinematic implementation of the High-Level Reaching benchmark that eliminates the need for a simulator entirely. Instead of executing motion commands on a physical robot model, the system performs direct forward kinematics (FK) computations to update a ”fake” end-effectors position in cartesian space. At each step, the agent chooses one of six discrete movement commands, each translating the end-effector in a positive or negative direction along one of the three coordinate axes. The new position of the end-effector is computed by applying a fixed offset in the chosen axes direction to the previous position. This results in a lightweight, fully deterministic environment that replicates the kinematic behavior of the simulated arm but omits physical effects such as inertia or collisions. The reward and termination criteria remain identical to those of the simulated benchmark, allowing direct comparability of results. Because no physics or message communication needs to be simulated, this version runs orders of magnitude faster (see Appendix D for a comparison), making it ideal for large-scale hyper-parameter searches. 4.5 Low-Level Reaching (LLR) In this novel benchmark, the agent controls the robot arm at the level of joint angles instead of cartesian coordinates. The goal remains to reach a given 3D target position with the end-effector, requiring coordinated multi-joint motion and precise spatial reasoning. To this end, an implicit model for local inverse kinematics is learnt around the initial configuration of the arm. An episode spans 7 steps, in each of which the agent selects a target angle for a single joint. The joints are controlled sequentially in a fixed order, where step s controls joint s . The action space remains discrete, with each action representing one out of 5 discrete angles in the joint’s allowed angle range. The benchmark consists of 8 tasks, each defined by a goal position located at varying locations and heights around the arm’s workspace. Observations include the current cartesian position of the end-effector, the cartesian goal position, the joint angles and the index of the joint being controlled. This additional index helps the agent specialize its output depending on which joint is active at each time step. Further details about observation and actions spaces are provided in Section B.5 . Due to the discrete episode structure and single final reward, we adopt a simplified policy gradient approach similar to REINFORCE, where only the final reward is used for updating the policy. In our setup, all intermediate rewards are set to 0.0 and only the final step receives a non-zero reward, with the discount factor \gamma=1.0 . This setup allows for rapid experimentation and emphasizes the learning of accurate motor programs over multiple joints without requiring dense feedback. For details of reward computation, see Section C.3 . 4.6 Kinematic Low-Level Reaching (LLR-K) Similarly to the HLR benchmark, the LLR benchmark includes a kinematic variant based entirely on forward and inverse kinematics (IK) calculations. Here, the robot arm is represented as a 7-d.o.f. kinematic chain without any physical simulation. The agent operates directly in joint space, selecting actions that modify joint angles. The resulting end-effector position is computed using forward kinematics, and the next observation is derived from the new joint and cartesian states. In contrast to the simulated setup, this version bypasses Gazebo and the Gazebo-Transport messaging layer entirely, drastically reducing computation time and hardware requirements. By isolating the kinematic problem, it allows efficient evaluation of learning dynamics, exploration strategies, and policy architectures without the overhead of physics simulation. The kinematic reaching benchmarks therefore complement their simulated counterparts: they preserve the same task definitions, reward functions, and termination conditions, but trade physical realism for computational efficiency, making them particularly well suited for rapid experimentation and algorithm development. 5 Experiments Our experiments were conducted on a High-Performance Computing (HPC) cluster consisting of 40 Linux workstations, each equipped with an Nvidia GTX 3090 graphics card. We used TensorFlow 2.14 as the primary deep learning framework to implement and train our reinforcement learning models. All experiments were deployed using Apptainer containers, providing a reproducible environment. Table 1 : Evaluation of randomly picked individual tasks in the MPO and MLF benchmarks for all settings. We observe a small degradation between SSS and DS, which may be explained by increased task difficulty. MPO results differ between tasks according to whether to target object should be pushed (+10 reward) or not pushed (only small reward for stopping in time). task \rightarrow 51 52 53 54 106 107 108 109 \downarrow setting MLF DS 1.45 1.48 1.44 1.48 1.49 1.44 1.42 1.41 SS 1.52 1.52 1.51 1.49 1.54 1.51 1.52 1.50 SSS 1.61 1.58 1.61 1.61 1.57 1.62 1.61 1.61 \downarrow setting MPO DS 21.0 11.3 22.7 12.0 21.5 11.5 21.5 12.0 SS 21.3 11.8 20.9 12.9 22.0 11.8 21.8 11.9 SSS 23.5 13.2 23.9 13.9 23.0 12.5 23.6 13.7 5.1 Multi-Task Line-Following (MLF) Experiments are conducted for each setting: default (DS), simplified (SS) and super-simplified (SSS), see Section 4.1 . The benchmark consists of 150 tasks, which are executed one after the other for 300 (100 for SS and SSS) episodes per task using 30 steps per episode. Epsilon-greedy exploration is used, decreasing \epsilon from either 1.0 (first task) or 0.5 (all other tasks) to a minimal value of 0.2 at the end of a task. A DNN with three hidden layers of size 100 maps observations to actions, where observations depend on the chosen setting, see Table 7 ). A replay buffer size of 15000 is used for all experiments, roughly 5 tasks’ worth of samples. For each task t^{\prime}