Classification of assembly tasks combining multiple primitive actions using Transformers and xLSTMs
Miguel Neves, Pedro Neto · 2025
Read Classification of assembly tasks combining multiple primitive actions using Transformers and xLSTMs by Miguel Neves, Pedro Neto (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-18012.txt.
Introduction
Miguel Neves and Pedro Neto sort long build tasks done by hand for robot team work in a 2025 paper. Task sorts help robots stay safe, call the next move, and learn from the hand. Long tasks that chain small acts balk at a cut into parts, which bars the plain sort route.
Hand joint tracks feed the sort in this work, with no cut to small acts. Three nets are set head to head on the same task runs: two known kinds and one fresh kin. Runs come from the trainer and from fresh hands.
Catalog record
| CREATOR | Miguel Neves, Pedro Neto |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 30 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-18012
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Classification of assembly tasks combining multiple primitive actions using Transformers and xLSTMs Miguel Neves, Pedro Neto Abstract The classification of human-performed assembly tasks is essential in collaborative robotics to ensure safety, anticipate robot actions, and facilitate robot learning. However, achieving reliable classification is challenging when segmenting tasks into smaller primitive actions is unfeasible, requiring us to classify long assembly tasks that encompass multiple primitive actions. In this study, we propose classifying long assembly sequential tasks based on hand landmark coordinates and compare the performance of two well-established classifiers, LSTM and Transformer, as well as a recent model, xLSTM. We used the HRC scenario proposed in the CT benchmark, which includes long assembly tasks that combine actions such as insertions, screw fastenings, and snap fittings. Testing was conducted using sequences gathered from both the human operator who performed the training sequences and three new operators. The testing results of real-padded sequences for the LSTM, Transformer, and xLSTM models was 72.9%, 95.0% and 93.2% for the training operator, and 43.5%, 54.3% and 60.8% for the new operators, respectively. The LSTM model clearly underperformed compared to the other two approaches. As expected, both the Transformer and xLSTM achieved satisfactory results for the operator they were trained on, though the xLSTM model demonstrated better generalization capabilities to new operators. The results clearly show that for this type of classification, the xLSTM model offers a slight edge over Transformers. K eywords LSTM \cdot Transformer \cdot xLSTM \cdot Assembly \cdot HRC 1 Introduction Figure 1: Training Pipeline. Extracted landmark coordinates are divided into sequences and padded according to their unique labels. Finally, the padded sequences are used to train the LSTM, Transformer, and xLSTM models. Advanced robotic systems must understand human actions to (1) learn collaboratively, (2) anticipate human needs, (3) ensure safety, and (4) adapt flexibly to increasingly unstructured environments [ 1 , 2 , 3 ] . In this context, the classification of human-performed assembly tasks is crucial for advancing collaborative robotics across all domains, particularly in manufacturing [ 4 , 5 , 6 ] . Most tasks performed by humans consist of sequential primitive actions, varying in length and complexity. While some tasks may seem trivial to classify using today’s methods, the classification of long-horizon tasks remains a significant challenge. The accuracy of human action classification depends on factors such as the length of the action sequence, as well as the number and similarity of the different primitive tasks it includes. Segmenting long tasks into smaller primitive actions is often challenging or unfeasible, highlighting the need for classifying relatively long tasks that encompass multiple primitive actions. For robots to recognize human actions, they require advanced perception and reasoning capabilities, supported by powerful machine learning models [ 7 , 8 , 9 ] . The recognition of human intent and the impact of robot movement on overall coordination are key factors [ 10 ] . An important criterion for the human-robot collaborative process is the signal timing used to deliver collaboration cues, as wrongly timed signals can lead to a deterioration in coordination [ 11 , 12 ] . A recent study proposes to solve long sequential tasks over long time horizons in robotic manipulation skills [ 13 ] . In the last two decades, a substantial number of classifiers for handling sequential data have relied on long short-term memory (LSTM) networks [ 14 ] , powered by data from different sensors to recognize gestures, motion primitives or primitive manufacturing tasks [ 15 , 16 , 17 ] . These recurrent networks have been widely studied and applied, achieving significant performance in classifying time series patterns. However, for longer sequences, these networks exhibit some performance deterioration. In 2017, Transformer networks revolutionized the field of artificial intelligence, significantly improving the performance of classification for longer pattern sequences [ 18 ] . Transformer-based networks have demonstrated highly promising results across diverse fields and have been shown to outperform LSTM-based solutions in tasks such as horse behaviour recognition, action recognition, and intent prediction [ 19 , 20 ] . However, a study on online trading has shown that LSTM networks may still outperform Transformer-based methods in certain tasks [ 21 ] . The success of Transformer-based networks in large language models (LLMs) has motivated their application to human action recognition [ 22 , 23 , 20 ] . Studies have also demonstrated promising results in applying Transformer-based networks to both action recognition and action anticipation tasks [ 24 , 25 ] . Additionally, other research has focused on recognizing various aspects of human-related information, such as intent [ 10 , 26 ] , state [ 7 ] , and task anticipation [ 27 ] . In 2024, the extended Long Short-Term Memory (xLSTM) architecture emerged as a promising alternative to Transformers for processing temporal data [ 28 ] . By introducing exponential gating and enhanced memory structures, xLSTM addresses limitations of traditional LSTMs, such as limited storage capacity and lack of parallelizability, thereby improving the modelling of long-term dependencies in sequential data. Recent results suggest that xLSTM-based models provide a competitive alternative to Transformer-based models for long-term time series forecasting tasks [ 29 ] . In this study, we propose classifying long assembly sequential tasks that include multiple primitive actions using hand landmark coordinate data from the CT benchmark dataset, Fig. 1 . The performance of two well-established classifiers, LSTMs and Transformers, is compared with the recent xLSTM model. Additionally, we propose a novel methodology for real-time labelling of different assemblies, reducing the human effort and time required to set up and operate the system. 2 Methodology 2.1 Long Short-Term Memory (LSTM) Recurrent neural networks (RNNs) were developed from standard feed forward neural networks (FFNN) by adding recurrent connections and a hidden state vector, enabling them to capture time-relationships within sequences of inputs. Though, vanilla RNNs often struggle in retaining long-term dependencies [ 30 ] . This limitation led to the development of LSTM networks [ 14 ] . LSTMs have two key components: the cell state c and the hidden state h . The cell state is the memory of the network, carrying information across different time steps, i.e., long-term memory, and the hidden state is the output of the LSTM cell for a given time step, i.e., short-term memory. These states are updated and managed using three gates: the forget gate, the input gate, and the output gate. The forget gate determines what information to discard from the previous cell state through the following: f_{t}=\sigma\left(\mathbf{w}_{f}^{\mathsf{T}}\mathbf{x}_{t}+r_{f}h_{t-1}+b_{f}\right) (1) Where f_{t} is the forget gate at time step t , \sigma is the Sigmoid activation function, \mathbf{w}_{f} is the weight vector between the input vector \mathbf{x}_{t} and the forget gate, h_{t-1} is the hidden state from the previous time step, r_{f} are the recurrent weights between the hidden state and the forget gate, and b_{f} is the bias vector for the forget gate. The input gate determines what information to add to the cell state. First a candidate state is calculated from: \tilde{c}_{t}=\textrm{tanh}\left(\mathbf{w}_{c}^{\mathsf{T}}\mathbf{x}_{t}+r_{c}h_{t-1}+b_{c}\right) (2) Where \tilde{c}_{t} is the candidate cell state, tanh is the hyperbolic tangent activation function, \mathbf{w}_{c} is the weight vector between the input \mathbf{x}_{t} and the candidate cell state, r_{c} are the recurrent weights between