On the capabilities of LLMs for classifying and segmenting time series of fruit picking motions into primitive actions
Eleni Konstantinidou, Nikolaos Kounalakis, Nikolaos Efstathopoulos, Dimitrios Papageorgiou · 2025
Read On the capabilities of LLMs for classifying and segmenting time series of fruit picking motions into primitive actions by Eleni Konstantinidou, Nikolaos Kounalakis, Nikolaos Efstathopoulos, Dimitrios Papageorgiou (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2507-07745.txt.
Introduction
Eleni Konstantinidou, Nikolaos Kounalakis, Nikolaos Efstathopoulos, and Dimitrios Papageorgiou test whether large language models can break fruit picking moves into basic acts. In learning by demonstration, a robot learns a job after a person shows it. Splitting a smooth move into acts like push, pull, and twist is a key step.
Instead of classic trained classifiers, the team asks language models to label and cut motion data streams. They compare three ways of tuning the models for the task. The motion data came from guiding a UR10e robot arm by hand through a fruit picking scene. The goal is a method that is easy to set up and use in real farm work.
Catalog record
| CREATOR | Eleni Konstantinidou, Nikolaos Kounalakis, Nikolaos Efstathopoulos, Dimitrios Papageorgiou |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 20 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2507-07745
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
On the capabilities of LLMs for classifying and segmenting time series of fruit picking motions into primitive actions Eleni Konstantinidou, Nikolaos Kounalakis, Nikolaos Efstathopoulos, Dimitrios Papageorgiou Abstract Despite their recent introduction to human society, Large Language Models (LLMs) have significantly affected the way we tackle mental challenges in our everyday lives. From optimizing our linguistic communication to assisting us in making important decisions, LLMs, such as ChatGPT, are notably reducing our cognitive load by gradually taking on an increasing share of our mental activities. In the context of Learning by Demonstration (LbD), classifying and segmenting complex motions into primitive actions, such as pushing, pulling, twisting etc, is considered to be a key-step towards encoding a task. In this work, we investigate the capabilities of LLMs to undertake this task, considering a finite set of predefined primitive actions found in fruit picking operations. By utilizing LLMs instead of simple supervised learning or analytic methods, we aim at making the method easily applicable and deployable in a real-life scenario. Three different fine-tuning approaches are investigated, compared on datasets captured kinesthetically, using a UR10e robot, during a fruit-picking scenario. This paper is a Late Breaking Results report and it will be presented through a poster at the 34th IEEE International Conference on Robot and Human Interactive Communication (ROMAN), 2025 at Eindhoven, the Netherlands. I Introduction Artificial Intelligence (AI) has become a powerful force that is revolutionizing various fields by developing systems that mimic human intelligence, encompassing this way areas like reasoning, learning and problem-solving [ 1 ] . This transformation is primarly due to Large Language Models (LLMs), deep learning models capable of processing and generating natural language data and other content [ 2 ] . The evolution of LLMs was driven by the growth of the internet and increasing computing power, which fuelled the vision of Natural Language Processing (NLP) [ 3 ] through deep learning methods such as Recurrent Neural Networks (RNNs) [ 4 ] , and the Transformer model [ 5 ] . This nourished models like BERT [ 6 ] and Generative Pre-Trained Transformer (GPT) [ 7 ] , revolutionizing fields like text generation and language translation. From a ”robotics” perspective, segmenting a complex motion into primitive actions is a crucial step in learning human kinematic behaviors through demonstrations. While many works employ Deep Neural Networks (DNN) [ 8 ] [ 9 ] for segmentation, and others propose analytic solutions [ 10 ] [ 11 ] or even AI methods like Temporal Convolutional Networks (TCN) [ 12 ] [ 13 ] for explicitly segmenting motion time-series, analytic methods require solid technical skills for action detection and classification, making them hard to deploy in a real-world scenario. This work investigates LLM capabilities for classifying primitive actions and segmenting complex fruit-picking motions. In particular, we explore and compare three LLM fine-tuning approaches: a) using only linguistic explanation of the primitive action, b) using a small dataset of representative example motions (e.g., five samples per action) and c) combining both a) and b) approaches. The utilized datasets for both the fine-tuning and experimental evaluation/comparison of the approaches, consists of kinesthetically captured data, i.e. motion time-series captured by having the human demonstrating the motion directly by moving the robot’s end-effector through physical contact. II Motivation and Problem Description The advantages of picking a fruit without utilizing cutting tools span from ensuring the safety of the plant and fruit, to being effective, as in many cases there are no cutting affordances for the insertion of a cutting tool (e.g. in the case of peaches). The complex motion required for detaching a fruit from the branch depends on the type of the plant. However, in most of the cases, this complex motion is composed by a sequence of simpler primitive actions. As we only account for the detachment of the fruit, these primitive actions are considered to be the following: Pull , Slide , Swing , Tilt , Twist , as presented in Fig. 1 , with either one of the first two ( Pull or Slide ) always appearing at the end of the complex detachment motion, as they involve a displacement of the fruit from the branch. Fig. 1 : The considered primitive actions To teach a fruit picking skill to a novice farmer, humans firstly utilize verbal communication, i.e. linguistic explanation. An example sentence provided by an experienced farmer to the novice one, to this aim, could be the following: ”To detach the fruit, you firstly have to twist it and then pull it from the branch.” . In most of the cases, the experienced farmer also provides an example of the fruit picking procedure, by demonstrating it in front of the learner. Inspired by the intuition characterizing the human-to-human skill transfer, in this work we aim at investigating the use of Large Language Models (LLMs) for segmenting a complex sequence of primitive actions, in the context of the fruit picking task. Our aim is to exploit these capabilities as a part of Learning by Demonstration (LbD) for a robotic system which will be capable to learn from humans through intuitive means. Let us assume the availability of a demonstrated fruit picking motion. Therefore, let \{F\} be the frame at the initial fruit’s pose and \mathbf{x}(t)\triangleq[\mathbf{p}^{\intercal}(t)\;\;\mathbf{Q}^{\intercal}(t)]^{\intercal}\in\mathbb{R}^{3}\times\mathbb{S}^{3} be the pose of the robot’s end-effector with respect to \{F\} , where \mathbf{p}(t)\in\mathbb{R}^{3} and \mathbf{Q}(t)\in\mathbb{S}^{3} are the end-effector’s position and orientation in unit quaternion form respectively, with \mathbb{S}^{3} denoting the unit sphere in \mathbb{R}^{4} . Moreover, let \mathbf{v}(t)\triangleq[\dot{\mathbf{p}}^{\intercal}(t)\;\;\boldsymbol{\omega}^{\intercal}(t)]^{\intercal}\in\mathbb{R}^{6} be the generalized velocity of the end-effector with respect to \{F\} , with \dot{\mathbf{p}}(t),\boldsymbol{\omega}(t)\in\mathbb{R}^{3} being its linear and angular velocity respectively. Given that \mathbf{x}(t) is provided by the forward kinematics of the robot, one can calculate a numerical approximate of \mathbf{v}(t) , based on the numerical differentiation of the pose, as described in the following section. Given a time series of the generalized velocity, \mathbf{v}(t),t\in[0,T] , of a complex fruit-picking motion, with respect to the initial pose of the fruit, with T\in\mathbb{R}_{>0} being the duration of motion, our aim is to test the effectiveness of an LLM to simultaneously classify and segment the motion into the aforementioned five predefined primitive actions, shown in Fig. 1 . Notice that the velocity of the end-effector is considered, instead of its pose, in order to facilitate the segmentation by rejecting bias. In other words, by selecting the velocity instead of the pose, each action maintains its characteristics even if it follows after a displacement that could possibly have occurred due to the sequential nature of the complex motion considered. III Considered approaches and comparison For classifying and segmenting the time series into primitive actions, a custom Generative Pre-trained Transformer (GPT) was created, based on OpenAI’s GPT-4-turbo model via the ChatGPT Custom GPT framework 1 1 1 https://help.openai.com/en/articles/8554397-creating-a-gpt . A key aspect of the configuration was the emphasis on rule-based classification, where the identification and segmentation of motions were grounded in the predefined kinematic characteristics listed in Table I , focusing on the dominant directional patterns observed in the translational and angular velocity components. Three fine-tuning (learning / explanation) approaches are considered for our case study. The first approach (Approach A) involves only linguistic explanation of the primitive actions. In particular,