Learning-Based Vision Systems for Semi-Autonomous Forklift Operation in Industrial Warehouse Environments
Vamshika Sutar 1 , Mahek Maheshwari 1 , Archak Mittal 1 2 · 2025
Read Learning-Based Vision Systems for Semi-Autonomous Forklift Operation in Industrial Warehouse Environments by Vamshika Sutar 1 , Mahek Maheshwari 1 , Archak Mittal 1 2 (2025) in the Tower of Babel, a free online library of public-domain works preserved in clean, searchable text. No sign-up. The full text is served as a plain-text file at a predictable URL, where text is present: https://antarctic-labs.com/tower-of-babel/arxiv-2511-06295.txt.
Introduction
Vamshika Sutar, Mahek Maheshwari, and Archak Mittal build a low-cost vision system to guide forklifts in warehouses. Automating material handling needs perception that is robust yet cheap enough to retrofit onto ordinary machines. Their framework uses a single standard camera to find pallets and the holes a forklift's forks must enter.
Two recent YOLO detector versions were tuned with automated hyperparameter search and spatial cleanup steps. A mapping module then turns raw detections into spatial layouts that pair each pallet with its holes for action. On a custom dataset enriched with real warehouse photos, one model led on raw detection accuracy while the tuned other model gave better precision and steadier training.
Catalog record
| CREATOR | Vamshika Sutar 1 , Mahek Maheshwari 1 , Archak Mittal 1 2 |
|---|---|
| YEAR | 2025 |
| COLLECTION | PAPERS |
| FORMAT | TXT |
| FILE SIZE | 40 KB |
| VERSION | 1.0 |
| LICENSE | http://creativecommons.org/licenses/by/4.0/ |
| RIGHTS | CC BY |
| SOURCE | arXiv |
Source: https://arxiv.org/abs/2511-06295
Browse the full library · About the Tower of Babel
Full text
The opening of the work, from the library's hosted text:
Learning-Based Vision Systems for Semi-Autonomous Forklift Operation in Industrial Warehouse Environments Vamshika Sutar 1 , Mahek Maheshwari 1 , Archak Mittal 1 2 Abstract The automation of material handling in warehouses increasingly relies on robust, low-cost perception systems for forklifts and Automated Guided Vehicles (AGVs). This work presents a vision-based framework for pallet and pallet-hole detection and mapping using a single standard camera. We utilized YOLOv8 and YOLOv11 architectures, enhanced through Optuna-driven hyperparameter optimization and spatial post-processing. An innovative pallet-hole mapping module converts the detections into actionable spatial representations, enabling accurate pallet–hole association for semi-autonomous forklift operation. Experiments on a custom dataset augmented with real warehouse imagery show that YOLOv8 achieves up to 95% pallet detection accuracy and 72% pallet-hole accuracy, while YOLOv11, particularly under optimized configurations, offers superior precision and stable convergence. The results demonstrate the feasibility of a cost-effective, retrofittable visual perception module for forklifts. This study proposes a scalable approach to advancing warehouse automation, promoting safer, economical, and intelligent logistics operations. I Introduction The warehousing and logistics sector is undergoing a rapid transformation driven by e-commerce growth and persistent labor shortages, with companies citing labor as a top challenge. This has accelerated investment in automation, with companies planning to evaluate Automated Guided Vehicles (AGVs) and Autonomous Mobile Robots (AMRs) in the next few years. As a cornerstone of this shift, AGVs and robotic forklifts are pivotal for automating material handling tasks like pallet transportation, stacking, and retrieval [ 1 ] . The linchpin of these operations is robust perception, the ability to accurately detect, localize, and precisely align with pallets and their insertion points (fork holes). This capability is fundamental not only for navigation but also for ensuring the safe and efficient physical interaction required for material handling. State-of-the-art solutions for this perceptual challenge have traditionally relied on sophisticated sensor suites. Methods using QR codes or high-resolution 3D LiDARs are common but introduce significant drawbacks, including high cost, sensitivity to viewing angles, and limitations in dynamic environments. While advanced vision-based methods like lightweight segmentation networks (LCS-Net) show promise for precise pallet localization using visual data [ 2 ] , achieving a balance between high accuracy, real-time inference speed, and generalization across diverse pallet types remains a significant challenge. Furthermore, the prospect of replacing entire fleets of manual forklifts with new, sensor-integrated autonomous models is often financially untenable, highlighting a critical gap in the availability of low-cost, retrofittable perception technologies [ 3 ] . To bridge this gap, we propose a vision-based framework for pallet and pallet-hole detection and mapping that relies exclusively on a single standard camera. Our approach leverages the latest advancements in efficient deep learning, specifically YOLOv8 and a next-generation YOLOv11 architecture, to achieve high-precision detection at real-time inference speeds. This design prioritizes cost-effectiveness and ease of integration, allowing for seamless retrofitting onto existing forklift fleets. Moving beyond mere detection, our second key contribution is a novel pallet-hole mapping module. This component transforms raw, 2D bounding box detections into actionable spatial representations within the vehicle’s operational framework. By accurately estimating the insertion points of pallets, this module provides the essential intelligence for AGVs to execute precise navigation, alignment, and successful fork insertion. This end-to-end framework delivers a robust, scalable, and practical perception system designed to accelerate the adoption of automation in real-world warehouse environments. II Related Work Research in warehouse automation has advanced through developments in computer vision and deep learning, focusing on enabling precise pallet handling for autonomous systems. Key areas include pallet detection and pallet hole detection, essential for navigation, stacking, and fork alignment. Early methods relied on handcrafted features and geometric models, which struggled in cluttered or dynamic environments. Recent approaches leverage CNN-based detectors, RGB-D sensing, and attention mechanisms to improve accuracy and speed. However, challenges remain in robust hole detection and mapping. The following sections review prior work in these areas, identifying gaps that motivate our proposed YOLO-based solution for integrated pallet and hole detection. II-A Pallet Detection Pallet detection is a fundamental capability in warehouse automation, as it enables forklifts and automated guided vehicles (AGVs) to localize load carriers for navigation, stacking, and retrieval. Early research was mainly focused on detecting the overall pallet structure from monocular RGB images using deep learning-based detectors. Xiao et al. [ 4 ] demonstrated pallet recognition and localization from RGB-D contours, but their system was limited to pallet body detection. Mohamed et al. [ 5 ] proposed a hybrid pallet detection and localization framework combining Faster R-CNN on RGB images with CNN-based classification of 2D rangefinder data, followed by Kalman filtering. Their system achieved 99.58% detection accuracy and \sim 2 cm localization error, though it did not address pallet hole detection, limiting fork insertion tasks. Kai et al. [ 6 ] proposed a CNN-based front-face shot framework to estimate pallet pose for forklift alignment. Hussain et al. [ 7 ] designed a lightweight CNN for detecting damaged pallet racks, optimized for real-time embedded deployment. While focused on rack integrity, their results highlight the potential of efficient CNNs for pallet-related perception. Keshri et al. [ 8 ] developed a YOLOX-based approach with a monocular RGB camera, using focal length calibration to estimate pallet coordinates and distances. The system achieved \sim 70% accuracy, but again lacked pallet hole detection, constraining automation precision. More recently, Kita et al. [ 9 ] leveraged 360^{\circ} image projections for pallet detection and 3D pose estimation, attaining \pm 1^{\circ} yaw error and \sim 2 cm position error, making it practical for forklift alignment with pallets. Gann et al. [ 10 ] investigated pallet detection using synthetic and domain-randomized datasets with YOLOv8, achieving significant improvements in detecting stacked and racked pallets under varied lighting. Overall, these studies demonstrate significant progress in pallet detection but remain limited to CNN-based or 2D/3D detection frameworks without explicit pallet hole recognition. In contrast, our work employs advanced YOLO models (YOLOv8, YOLOv11) to achieve both pallet and pallet hole detection, along with pallet-hole mapping for AGV operations, enabling a more complete solution for warehouse automation. II-B Pallet hole detection Pallet detection, pallet hole detection, and the reliable mapping of pallet holes to their corresponding pallets are essential for enabling automated forklifts to accurately position forks for load handling. Early research by Cucchiara et al. [ 11 ] introduced a model-based method that combined Hough transforms with virtual corner features to identify pallets and approximate pallet holes, providing a foundation for fork guidance. However, such handcrafted approaches lacked robustness in cluttered and dynamic warehouse environments. Building on advances in sensing, Xiao et al. [ 4 ] proposed an RGB-D based system that detected pallet structures and holes using geometric constraints, enabling fork insertion but with degraded performance under poor lighting and occlusion. More recently, Wang and Bo [ 12 ] enhanced YOLOv5 with Efficient Channel Attention (ECA) and Ghost-Shuffle Convolution (GSC) to improve pallet hole recognition in RGB-D images, achieving higher accuracy and real-time performance, yet their method did not establish explicit correspondence between detected holes and pallets. Addressing this gap, our approach introduces a centroid- and IoU-based mapping strategy that explicitly links pallet holes to their respective pallets, ensuring robust and precise fork alignment in real-world warehouse environments. III Dataset The data pipeline includes a custom dataset