Self-Supervised Learning for Robotic Leaf Manipulation: A Hybrid Geometric-Neural Approach
Srecharan Selvam, Abhisesh Silwal, George Kantor · 2025
Read Self-Supervised Learning for Robotic Leaf Manipulation: A Hybrid Geometric-Neural Approach by Srecharan Selvam, Abhisesh Silwal, George Kantor (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2505-03702.txt.
Introduction
Srecharan Selvam and co-authors teach a robot arm to grasp leaves gently in farm settings. Their 2025 paper joins classic geometry with neural nets, since leaves bend and vary too much for either alone. A vision model spots each leaf, a stereo method reads depth, and two scoring paths then pick the grasp point.
A trust-weighted blend merges the geometry score with the neural score by how sure each one is. The geometry path also acts as teacher, making training labels on its own. Trials hit eighty-eight percent success in lab terms and near eighty-five in a real greenhouse. Both beat pure geometry and pure neural runs by clear margins.
Catalog record
| CREATOR | Srecharan Selvam, Abhisesh Silwal, George Kantor |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 52 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2505-03702
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Self-Supervised Learning for Robotic Leaf Manipulation: A Hybrid Geometric-Neural Approach Srecharan Selvam, Abhisesh Silwal, George Kantor Abstract Automating leaf manipulation in agricultural settings faces significant challenges, including the variability of plant morphologies and deformable leaves. We propose a novel hybrid geometric-neural approach for autonomous leaf grasping that combines classical computer vision with neural networks through self-supervised learning. Our method integrates YOLOv8 for instance segmentation and RAFT-Stereo for 3D depth estimation to build rich leaf representations, which feed into both a geometric feature scoring pipeline and a neural refinement module (GraspPointCNN). The key innovation is our confidence-weighted fusion mechanism that dynamically balances the contribution of each approach based on prediction certainty. Our self-supervised framework uses the geometric pipeline as an expert teacher to automatically generate training data. Experiments demonstrate that our approach achieves an 88.0% success rate in controlled environments and 84.7% in real greenhouse conditions, significantly outperforming both purely geometric (75.3%) and neural (60.2%) methods. This work establishes a new paradigm for agricultural robotics where domain expertise is seamlessly integrated with machine learning capabilities, providing a foundation for fully automated crop monitoring systems. 1 Introduction Agricultural robotics has emerged as a key technology for addressing labor shortages and improving efficiency in modern farming [ 8 , 43 ] . Among greenhouse cultivation tasks, leaf sampling for disease detection remains a significant bottleneck, requiring skilled workers to manually identify, select, and extract tissue samples from thousands of plants [ 42 , 4 ] . This labor-intensive process increases operational costs and limits the frequency of plant health monitoring, potentially allowing diseases to spread undetected [ 5 , 33 ] . Automating leaf manipulation presents unique challenges compared to traditional robotic grasping tasks. Unlike rigid industrial objects, plant leaves are deformable, vary significantly in size and orientation, and are often partially occluded in dense canopies [ 30 , 38 ] . While recent advances in deep learning have revolutionized robotic grasping for industrial applications [ 34 , 37 , 53 ] , these approaches typically require large datasets of labeled grasp points—a resource that is prohibitively expensive to create for agricultural settings where plant morphology varies continuously throughout growth cycles [ 26 ] . Existing approaches to agricultural manipulation fall into two categories: purely geometric methods that rely on hand-crafted features [ 18 , 44 , 20 ] , and end-to-end deep learning systems trained on synthetic or limited real-world data [ 3 , 52 , 49 ] . Geometric approaches, while interpretable and robust to domain shifts, struggle with the natural variability of plant structures. Conversely, deep learning methods excel at handling complex visual patterns but suffer from poor generalization when deployed on new crop varieties or growth stages not represented in their training data [ 36 ] . We present a novel hybrid approach that leverages the complementary strengths of geometric reasoning and neural networks through self-supervised learning. Our key insight is that traditional computer vision algorithms, despite their limitations, encode valuable domain expertise that can serve as a teacher for training neural networks without manual annotation [ 21 ] . This approach enables continuous learning from operational data while maintaining the interpretability and reliability required for agricultural automation. Our system operates on a 6-DOF gantry robot equipped with stereo vision and a custom end-effector for leaf manipulation. The perception pipeline combines YOLOv8 instance segmentation [ 23 ] with RAFT-Stereo depth estimation [ 31 ] to generate 3D representations of plant canopies. For grasp point selection, we implement a dual-path architecture: a geometric pipeline using Pareto optimization across multiple hand-crafted features (flatness, accessibility, edge distance), and a convolutional neural network with spatial attention that learns from the geometric system’s decisions [ 25 ] . The main contributions of this work include: • A self-supervised learning framework where geometric algorithms act as expert teachers for neural networks, eliminating the need for manual grasp annotation in agricultural settings • A hybrid decision architecture that dynamically weighs geometric and learned features based on prediction confidence, achieving robust performance across diverse plant conditions • A comprehensive grasp point selection system incorporating novel scoring functions tailored to leaf-specific constraints such as deformability, approach angles, and occlusion handling • Extensive validation on thousands of real plant samples demonstrating significant improvements over traditional geometric methods, particularly in challenging scenarios with partial occlusion and irregular orientations This work provides a foundation for fully automated crop monitoring systems and establishes a new paradigm for agricultural robotics where domain expertise is seamlessly integrated with machine learning capabilities. 2 Related Work 2.1 Vision-Based Leaf Manipulation Traditional approaches to robotic leaf manipulation in agricultural settings relied on geometric reasoning and classical computer vision. Hemming et al. developed methods for cucumber leaf detection in greenhouses using color and texture features [ 19 ] , while Bac et al. presented obstacle-aware motion planning for tomato canopies [ 6 ] . Several studies focused on deformable leaf modeling, including Cerutti et al.’s parametric active polygon models [ 9 ] , Xia et al.’s active shape models for overlapping leaves [ 51 ] , and Jin et al.’s probabilistic graphical models for leaf structure analysis [ 22 ] . The integration of 3D information improved robustness, as demonstrated by Guo and Xu’s multiview stereo reconstruction for lettuce segmentation [ 16 ] and Sodhi et al.’s plant growth monitoring system using structure-from-motion [ 45 ] . While effective in controlled conditions, these methods often required extensive tuning and struggled with natural plant variability, particularly under varying illumination conditions [ 11 ] . 2.2 Deep Learning for Agricultural Grasping Deep learning has shown promise in agricultural manipulation, though with unique challenges compared to industrial applications. Barth et al. developed CNN-based systems for broccoli harvesting that handle significant occlusion [ 7 ] , while Arad et al. demonstrated sweet pepper harvesting combining YOLO detection with stereo depth [ 3 ] . For leaf-specific tasks, Ahlin et al. pioneered CNN-based leaf identification with visual servoing for autonomous sampling, achieving 85% success rates in greenhouses [ 2 ] . However, these approaches typically require extensive training data—a significant limitation given the continuous variation in plant morphology [ 47 ] . To address this, researchers have explored simulation, with approaches like Dex-Net generating synthetic grasp scenarios [ 34 ] , inspiring agricultural adaptations for data generation [ 28 ] . Recent advances in generative data augmentation have shown promise for bridging the synthetic-real domain gap [ 48 ] , particularly for multi-crop environments with challenging lighting conditions. 2.3 Self-Supervised Learning in Agricultural Robotics Self-supervised learning has emerged as a promising paradigm for agricultural robotics, particularly where manual annotation is expensive. Zhang et al. demonstrated self-supervised learning for tomato harvesting, using classical vision systems to provide training labels [ 54 ] . Similar bootstrapping approaches include Kootstra et al.’s work on sweet pepper detection, where geometric algorithms generated training data for neural networks [ 27 ] and Tao et al.’s approach using temporal consistency for plant growth tracking [ 46 ] . This knowledge transfer from classical to learning-based systems has proven particularly valuable in controlled environment agriculture, where hybrid approaches consistently outperform purely learned policies [ 42 , 14 ] . Recent work has also explored contrastive learning frameworks for agricultural visual