ATLAS: An A nnotation T ool for L ong-horizon Robotic A ction S egmentation
Sergej Stanovcic, Daniel Sliwowski, Dongheui Lee · 2026
Read ATLAS: An A nnotation T ool for L ong-horizon Robotic A ction S egmentation by Sergej Stanovcic, Daniel Sliwowski, Dongheui Lee (2026) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2604-26637.txt.
Introduction
Stanovcic, Sliwowski, and Lee introduced ATLAS, a tool for long-horizon robotic action segmentation. Existing tools favor vision-only data, lack synchronized robot time-series views, or resist new dataset formats. ATLAS synchronizes multi-view video with proprioceptive signals, annotates boundaries, labels, and outcomes, and reads ROS bags and RLDS formats through a modular layer. Its keyboard-centric design cut annotation time per action by at least 6 percent versus ELAN, improved expert alignment nearly 3 percent with time-series data, and cut boundary error fivefold against vision-only tools.
Catalog record
| CREATOR | Sergej Stanovcic, Daniel Sliwowski, Dongheui Lee |
|---|---|
| YEAR | 2026 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30405 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2604.26637
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
ATLAS: An A nnotation T ool for L ong-horizon Robotic A ction S egmentation Sergej Stanovcic, Daniel Sliwowski, Dongheui Lee Abstract Annotating long-horizon robotic demonstrations with precise temporal action boundaries is crucial for training and evaluating action segmentation and manipulation policy learning methods. Existing annotation tools, however, are often limited: they are designed primarily for vision-only data, do not natively support synchronized visualization of robot-specific time-series signals (e.g., gripper state or force/torque), or require substantial effort to adapt to different dataset formats. In this paper, we introduce ATLAS , an annotation tool tailored for long-horizon robotic action segmentation. ATLAS provides time-synchronized visualization of multi-modal robotic data, including multi-view video and proprioceptive signals, and supports annotation of action boundaries, action labels, and task outcomes. The tool natively handles widely used robotics dataset formats such as ROS bags and the Reinforcement Learning Dataset (RLDS) format [ 12 ] , and provides direct support for specific datasets such as REASSEMBLE [ 13 ] . ATLAS can be easily extended to new formats via a modular dataset abstraction layer. Its keyboard-centric interface minimizes annotation effort and improves efficiency. In experiments on a contact-rich assembly task, ATLAS reduced the average per-action annotation time by at least 6% compared to ELAN [ 18 ] , while the inclusion of time-series data improved temporal alignment with expert annotations by more than 2.8% and decreased boundary error fivefold compared to vision-only annotation tools. I Introduction Annotated data plays a central role in many machine learning pipelines, as it is required either for model training, evaluation, or both. In supervised learning, annotations are directly used during training to compute the loss. While unsupervised methods do not require annotations at training time, in cases such as unsupervised action segmentation, they still rely on annotated data for quantitative evaluation [ 9 ] . As a result, the development and evaluation of both supervised and unsupervised models ultimately depends on the availability of annotated datasets. Creating such annotations typically requires dedicated software that allows human annotators to mark relevant information. Over the years, many tools have been developed for computer vision tasks such as object detection, semantic segmentation, and object tracking, including CVAT [ 1 ] and LabelMe [ 17 ] . However, these tools do not support the annotation of temporal action boundaries or action outcomes. Action-level annotations are particularly important in robotics, where long-horizon tasks can be decomposed into a sequence of simpler and shorter actions. Prior work has shown that explicit task decomposition is beneficial for learning long-horizon robotic policies [ 15 ] , and that decomposed demonstrations enable robots to learn high-level plans for complex tasks [ 3 ] . This motivates the need for tools that support precise temporal action annotation in robotic datasets. Fig. 1 : ATLAS is a tool for long-horizon robotic action segmentation, supporting time-synchronized visualization of multi-modal data. It handles common robotics dataset formats and can be extended to new ones (dashed line). The tool allows annotation of action boundaries, classes, and outcomes (success or failure). TABLE I : Comparison of existing annotation tools. ( \checkmark = Supported, \times = Not Supported) Time-series Temporal Supported Dataset Application Domain Visualization Segmentation Annotation Formats CVAT [ 1 ] Computer Vision \times \times Frames, Videos Foxglove Studio [ 4 ] Robotics \checkmark \times MCAP [ 5 ] , ROS Bags ROSAnnotator [ 19 ] Robotics \times 1 \checkmark ROS Bags Anvil [ 8 ] Behavioral \checkmark \checkmark Videos, Motion Capture ELAN [ 18 ] Linguistics \checkmark \checkmark Videos, CSV files Ours Robotics \checkmark \checkmark Frames, Videos, ROS Bags, RLDS, REASSEMBLE 1 The authors provide guidelines on how to modify the code to add support; however, this functionality is not implemented in the default version. Several existing tools, such as ELAN [ 18 ] and Anvil [ 8 ] , allow for annotating actions and their temporal boundaries and are widely used in behavioral and linguistic analysis. However, these tools are designed primarily for human-centered data and typically support video, audio, and motion capture visualization. They lack visualization of robot-specific signals such as end-effector pose, gripper state, or force and torque measurements, which are often critical for accurately identifying action boundaries, as not all events are clearly observable from vision alone [ 14 ] . Other tools, such as Foxglove Studio [ 4 ] , provide visualization capabilities for a wide range of robotic data types. However, they are intended for data inspection and do not support annotation creation. The closest existing annotation tool to our approach is ROSAnnotator [ 19 ] , which allows for annotating ROS bag files for human-robot interaction. While it supports camera visualization and audio transcription, it does not directly handle additional robotic data modalities. Extending support for such data requires manual code modifications, as noted by the authors. An additional challenge arises from the diversity of dataset formats used in robotics and related fields. Existing datasets cover a wide range of representations, including raw video files, ROS bag recordings, task-specific formats such as REASSEMBLE [ 13 ] , and more general data schemas such as the Reinforcement Learning Datasets (RLDS) format [ 12 ] , which is commonly used in robotics. Supporting multiple dataset formats, or allowing users to easily add custom dataset formats to the tool, enables the annotation of a broad range of already existing datasets. Adding additional annotations to existing datasets is particularly important for robotics, where collecting new data is expensive and time-consuming, and where valuable datasets already exist but may lack detailed annotations. In this work, we propose ATLAS , an annotation tool for long-horizon robotic action segmentation. The design of ATLAS is guided by several key criteria motivated by the practical challenges of annotating robotic datasets. First, the tool must support the visualization of time-synchronized data from multiple robotic sensors. This includes, but is not limited to, multi-view camera streams, gripper state, and end-effector pose, which together provide critical context for accurately identifying action boundaries and outcomes. Second, the tool is designed to natively support a wide range of existing dataset formats. In its current version, ATLAS supports raw videos, frame-based datasets, ROS bag files, the REASSEMBLE format, and the Reinforcement Learning Datasets (RLDS) format. Support for RLDS is particularly important, as it enables ATLAS to be applied to many existing robotics datasets without conversion. For example, all datasets in the Open X-Embodiment repository [ 10 ] are distributed in RLDS format, which currently includes 71 datasets comprising a total of 2,419,193 episodes. Third, ATLAS provides an abstract interface for integrating new datasets, allowing new data formats to be added with minimal effort. This abstraction enables the tool to scale beyond the formats currently supported and facilitates annotation of future datasets. Finally, the tool is designed to minimize annotation time by reducing interactions with the mouse and maximizing keyboard-based control. ATLAS supports customizable keyboard shortcuts, allowing users to adapt the interface to their annotation workflow and improve efficiency. In summary, ATLAS provides the following capabilities: • Visualization of time-synchronized, multi-modal robotic data, including multi-view RGB videos, gripper state, and end-effector pose. • Native support for multiple dataset formats, including videos, frame-based datasets, ROS bag files, and RLDS, as well as specific robotic datasets such as