An algorithm for drug retrieval based on robot-grasping detection constraints and DDPG autonomous learning Authors: Xiaowen Zhang, Tiegang Lv, Andrea Tigrini, Andrea Tigrini, Andrea Tigrini Published: 2025 ABSTRACT When the medicine-picking robot grasps drugs, its flexibility and accuracy in grasping detection mainly depend on the precision of visual guidance for the robot. The result of grasping detection directly determines whether the grasping task can be successfully completed. This study aims to enable a faster learning speed for the robot, reduce the search space for the grasping pose of the medicine-picking robot, and improve the grasping accuracy of the robot in unstructured environments. For this purpose, a self-learning DDPG grasping algorithm based on detection constraints is proposed and applied in automated pharmacy detection. The algorithm primarily consists of two steps. First, it extracts candidate grasping areas by analyzing the boundaries of the medicine. Second, with the aid of deep reinforcement learning, it inputs images with candidate grasping areas into an autonomous learning network, conducts adaptive noise exploration and perturbation in the search space, detects the optimal grasping point of the medicine from the image in real time, feeds it back to the medicine-fetching robot, adjusts the grasping pose through autonomous learning, and controls the robot to complete the training grasping. Experiments demonstrate that this method achieves a minimum of 15% improvement in grab detection accuracy compared with the four other grab detection methods. Within the confidence interval, it can achieve a grab success rate of 95%, which verifies the feasibility and effectiveness of this method. Introduction Automated pharmacies not only innovate existing hospital pharmacy management but also accelerate work efficiency, reduce the workload of pharmacists, strictly supervise and control the circulation process of drugs, ensure medication safety, and conform to the characteristics of modern pharmacy management system safety, science, and efficiency [1]. The sorting and distribution of drugs can be completed when dispensing robots are used to locate and grab drugs. The factors that affect the positioning, grasping, and detection accuracy of the drug retrieval robot include the end effector pose solved by the robot kinematics; the imaging model and calibration of various parameters of the vision system; and the target detection and grasping point pose. These factors directly affect the success rate of drug-grasping action. The most direct approach for ensuring accuracy in a medication-dispensing robot is to compensate for the time cost, which implies a competitive advantage over manual labor but is more difficult when ensuring a balance between accuracy and real-time performance. Furthermore, pharmacy automation requires robots to have autonomous learning capabilities for detecting the pose information of the target drug-grasping point during operation. This scheme can substantially improve the degree of pharmacy automation, such as drug grasping accuracy and real-time drug positioning. When the drug retrieval robot performs grabbing tasks, the flexibility and accuracy of grasping detection depend mainly on the precision of the visual guidance provided by the robot. The results of grasping detection directly determine whether the grasping task can be successfully completed. Grasping detection collects environmental image information via visual sensors before processing is completed, and these tasks are pure machine vision problems. For the drug retrieval robot to complete grasping, the task must be analyzed and planned using relevant knowledge such as kinematics, control theory, and machine vision. However, when the dispensing robot operates according to the predetermined plan, the position of its end gripper always deviates, and errors ultimately accumulate. Therefore, the current position of the gripper must be judged in terms of its suitability for gripping. This judgment is subsequently used as a basis to guide the dispensing robot to adjust the gripping posture. This study aims to enable a faster learning speed for robots, without relying on precise object and environmental information. The other aim is to quickly generate the grasping pose of the target drug based on the detection results of the drug detection algorithm, which satisfies the real-time requirements of the grasping operation. Thus, a grasping point detection algorithm based on DDPG autonomous learning is proposed and applied in automated pharmacies. This method, based on the results of object detection, generates grabbable areas and selects the location most likely to succeed in grabbing from these candidate areas. Then, it returns it to the medication retrieval robot. The process mainly consists of two steps. First, target objects are obtained from the target detection result image, and candidate grasping regions are extracted. These candidate regions are generated by analyzing the boundaries of the objects using edge detection operators. Then, these candidate regions are detected using deep reinforcement learning methods. By introducing adaptive noise perturbation into the exploration space, the optimal grasping box for the medicine is autonomously learned and detected from the image in real time. The graspable candidate regions are subsequently discriminated, which completes the mapping from the image to the grasping pose. The rest of the paper is organized as follows. The second part organizes relevant literature, conducts theoretical comparisons with existing methods, and introduces traditional reinforcement learning methods. The third part proposes an improved version of reinforcement learning based on exploration noise. The fourth part evaluates this method through extensive experiments and applies it to the grasping of a medicine-fetching robot. Finally, the fifth part summarizes the study. Related work The grasping planning of robots involves finding a suitable pose from an infinite number of possibilities to achieve stable and effective grasping. Bohg [2] divided robot-grasping planning into two categories: analysis-based grasping planning and data-based grasping planning. Prior to the 21st century, research mainly used analysis-based grasping planning methods, which formulated grasping planning problems as optimization problems under multiple constraints [3,4]. Contact mathematical models for the grasping process are established, grasping constraints and objective functions are set, and stable grasping poses are obtained via optimization solutions. The analysis-based grasping planning method is computationally complex and assumes that the surface characteristics, weights and weight distributions, and geometric and physical models of the object are accurately known. Precise modeling of the grasped object is required during grasping. However, accurate models for all objects in complex real-world environments are impossible to establish, and real robots and sensor systems inevitably entail errors. These gaps limit the planning and application of analysis-based grasping in real-world environments. In addition, considering the complexity of the models of some robotic arms and the objects to be grasped, the computational complexity of solving the grasping pose is extremely high, and real-time grasping operation requirements are difficult to meet [5]. In recent years, with breakthroughs in deep learning in the field of image processing, researchers have applied deep learning to robot-grasping planning [6–8]. Deep networks can automatically learn high-quality grasping image features from a large amount of image data. Compared with the method of manually designing grasping features, the performance of newer models has been greatly improved, thereby advancing the grasping planning method for unknown objects. Deep learning-based grasping planning methods can predict feasible grasping poses in the images collected by sensors via neural networks and rank the candidate grasping poses by using evaluation indicators, after which the best grasping pose is determined [9]. For the grasping planning method of sampling candidate grasping poses, candidate grasping pose generation and optimal grasping pose selection are two key steps that directly affect the grasping efficiency and success rate. In 2015, Lenz [10] designed a two-step deep network cascade system that uses a relatively small deep learning framework to detect several high-quality grasping regions in RGB-D images. Once the detection results are output to a larger deep network, the optimal grasping pose can also be determined. In 2016, Johns [11] designed a convolutional neural network (CNN) architecture in which the input and output of the network are used as the evaluation scores of the depth image and all candidate grasping poses on the image, respectively. In 2018, Morrison [12] proposed a small CNN that does not rely on sliding windows or bounding boxes and instead directly generates grasping poses and grasping quality on depth image pixels. Compared with other networks, CNNs have fewer parameters of several orders of magnitude, which can speed up grasping pose detection. However, the greatest drawback of deep learning-based grasping planning methods is the difficulty in balancing detection accuracy and real-time performance. The primary task of robot-grasping planning is to generate target-grasping candidate areas, which are generated on the basis of the external contour of the grasped drug. DRL networks are used for training to rapidly generate the optimal grasping area in these areas. In addition, DRL [13] refers to reinforcement learning combined with deep learning, which is an important branch of machine learning; it is a self-supervised learning method that has both the powerful perception ability of deep learning and the decision-making ability of reinforcement learning. Reinforcement learning [14] can describe and solve the problem of robot learning strategies, thereby maximizing returns or achieving specific goals during their interaction with the environment. Different reward functions are defined for different tasks, and the strategy network is optimized by continuously maximizing the reward value obtained at each step during the learning process. The DDPG [15]is a commonly used DRL method for solving high-dimensional continuous state space tasks. The DDPG is based on the actor–critic learning framework and utilizes the deterministic policy gradient method, fully combining the advantages of the Q-learning and policy gradient methods. DDPGs not only have unique policy characteristics of Q-learning that can fully utilize sampling data to reduce the number of robot trainings but also have the characteristics of the policy gradient method, which makes it applicable to high-dimensional continuous state space tasks. The algorithm framework is shown in Fig 1. Block diagram of the DDPG algorithm. Reinforcement learning is an iterative process that solves two problems at each iteration. Given a policy evaluation function, the policy is updated on the basis of the value function. The control strategy µ is the grasping strategy of the robot, and the Q network represents the action value function. In the reinforcement learning training phase, the control strategy obtains the current state si and outputs the action instructions ai of the robot. After the robot executes the action, the state of the operated object changes. This changed state is denoted by st+1. The reward value rt is calculated on the basis of the reward function. The robot stores the abovementioned four attributes as a tuple (st, at, st+1,rt) in its experience pool. When the amount of data stored in the experience pool reaches a certain level, the Q network is trained on uniformly sampled data from the pool. After training the Q network, the deterministic policy gradient method updates the robot control strategy. The robot continuously repeats this process until the control strategy converges. On the theoretical foundation of deep reinforcement learning, many researchers have begun to devote efforts to studying its application in robotic grasping detection. Zeng et al. [16] successfully completed the grasping and pushing actions of a robotic arm in an unstructured environment, in virtual settings and with real robots. In their study, they utilized observations from an RGB-D camera as input, integrated grasping and pushing strategies, and employed a Q-network to compute the maximum Q-values for these actions, which facilitated optimal action selection. Based on an improved DDPG algorithm. Dmitry Kalashnikov [17] from Google Brain adopted a scalable self-supervised learning method based on reinforcement learning, which is called QT-Opt. This method conducts grasping training on robots. During the training process, objects that are grasped receive a reward of 1, while objects that are not grasped receive a reward of 0. The collected data are used to train a deep reinforcement Q function, which evaluates the grasping position and orientation. Based on the evaluation values, the optimal grasping position and orientation are selected. After training 580,000 steps on robots, the success rate of grasping unknown objects reached 96%. To address the issue of the large number of training steps required by deep reinforcement learning algorithms, multiple robots are used to simultaneously train on grasping tasks to shorten training time. Breyer et al. [18] compared various robotic arm grasping methods based on reinforcement learning and proposed a deep reinforcement learning grasping method based on an action offset strategy. This method first uses an encoder–decoder network to reconstruct the depth image, which generates a depth prediction map containing embedded vectors to enhance the expressive power of features. Then, the embedded vectors are concatenated with the robotic arm state vectors, and a reinforcement learning policy network is used to predict the action offset. Finally, the robotic arm performs the corresponding grasping action based on the offset output by the network. Zhu [19] proposed an experience pool sampling strategy based on reward value priority and a method based on adaptive target exploration domain. Experiments conducted in a simulated environment for robotic arm grasping training showed improvements in terms of time required for convergence, number of steps needed for convergence, and convergence stability. Zhang [20] proposed a non-rational angle constraint strategy that dynamically evaluates the grasping effectiveness in different directions and imposes varying degrees of constraints on grasping actions in non-rational directions, which enhances the learning efficiency and grasping success rate of target grasping. By introducing the constraint factor in the non-rational angle constraint strategy, a grasping reward function with angle information is proposed to guide the grasping network to learn more accurate and detailed grasping strategies. To address the problem of sparse rewards during target grasping training, a grasping training method based on post-experience replay is adopted, which effectively improves the efficiency of target grasping training. You Tianya [21] proposed a deep reinforcement learning algorithm based on an improved Actor–Critic framework, which utilizes pushing actions to change the state of the environment and provide operational space for grasping when necessary. In scenarios with dense target objects, the algorithm achieves varying degrees of improvement in grasping success rate, action efficiency, and task completion rate. The grasping method based on deep reinforcement learning conducts grasping training through self-supervision, which continuously interacts with the environment to master stable grasping skills. This method does not require manually creating a large amount of labeled data to supervise model learning and exhibits good environmental adaptability. However, current grasping methods based on deep reinforcement learning often suffer from issues such as low model training efficiency and slow convergence speed. The literature [22] also points out that in engineering, the focus should be on providing higher autonomy for robotic systems. Therefore, more effective action strategies, learning methods, or reward mechanisms need to be introduced for faster grasping skills of the robotic arm. Proposed method This section primarily introduces our proposed DDPG autonomous learning grasping detection method for medicine-fetching robots based on detection constraints. In this method, candidate grasping areas are generated using target detection results to enhance the accuracy of grasping poses. The boundaries of the generated targets are also calculated and analyzed. Under the constraints of detection results, an autonomous learning grasping strategy network is designed, which outputs a reference rectangular box for grasping targets. Through action function mapping, the grasping pose is obtained. An adaptive exploration noise strategy is designed for perturbation to achieve broader exploration and autonomous learning within the grasping space of the robot. The details are elaborated below. Candidate capture area generation Given that this study primarily focuses on the grasping of boxed drugs, the chosen end-effector method is suction-based. The pose for grasping the drug is usually the center position of the medicine box. According to the YOLOv3 object detection algorithm output, the rectangular box of the drug encompasses the entire drug. However, some errors may exist between the output rectangular box and the actual boundary of the drug on the captured image. Candidate grasping areas are generated based on the boundary features of the drug to obtain an accurate grasping pose. An autonomous learning network is used to detect the suggested areas, and the optimal position is finally selected and controlled for grasping. This process is represented mathematically as follows: where R represents the candidate grasping area generated after drug detection, r is the possible grasping position within the candidate grasping area, g is the optimal grasping pose, and N denotes the mapping function corresponding to the autonomous learning network. Based on the constraints of force closure and shape closure, a rectangular region is set up to more accurately describe the grasping pose of the end effector and the spatial position of the medicine. This area is represented by a 5-tuple vector, as shown in the Fig 2. Schematic diagram of the 5-tuple of the grab area. The captured area in the image is represented by a 5-tuple as (x,y,w,h,θ). Within the rectangular area, the long side of the red rectangular box indicates the width of the medicine box w, the long side of the blue rectangular box indicates the length of the medicine box h, (x,y) corresponds to the coordinate value of the upper left corner of the positive direction rectangle of the medicine box, θ describes the direction of the rectangle, and the black box represents the ideal adsorption area of the end effector. From the results of drug detection, the (x,y) coordinates corresponding to the points on the plane can be easily determined. By utilizing depth information to determine the z coordinate, the grasping area can be represented as (x,y,z,w,h,θ). This 6-tuple describes the candidate area for the medicine-picking robot to grasp the medicine. The purpose of candidate grab regions is to determine the precise location of the medicine. It involves dividing the panoramic image into local regions, followed by further judgment to avoid redundant work. After the medicine detection is completed, the regions containing medicine are used as the search space for candidate regions. Generating candidate grab regions in the vicinity of the medicine detection rectangular box can significantly reduce computational workload and improve grab time. The optimal representation of the candidate region is when the rectangular box overlaps with the boundary box of the medicine, which can contain the complete medicine and eliminate the redundant background. This area can serve as the search space for the grasping pose of the medicine-picking robot. Based on prior experience, a set of potential grab regions is extracted at the boundaries of the drug to balance computational load and window traversal times. First, the boundaries of the drugs in the image are extracted; then, candidate regions are generated along the boundary curves in a clockwise direction, as shown in Fig 3. Starting from the top left pixel on the curve, the coordinates of the top left corner of the candidate region box are determined at intervals of 10 pixels. The normal direction of the fitted line of the set of these 10 points is used as the angle between the candidate region and the horizontal direction. Combining relevant knowledge of the rotation matrix, the candidate region including the boundary can be obtained and the region size can be determined using Equation (2). Schematic diagram of candidate region generation. Network design of the self-learning DDPG capture strategy Given that the geometric and physical information of the captured drugs is unknown, a specific strategy function or robot action library cannot be easily used to complete the robot-grasping task. The characteristics of the current drugs must be understood on the basis of sensor information to enable the drug retrieval robot to grasp drugs in unstructured environments. Autonomous learning actions for drug retrieval on the basis of these characteristics must also be planned accordingly. Drawing on the powerful image feature extraction ability of deep CNNs and combining it with reinforcement learning self-learning ability, a strategy network for drug retrieval robots to grasp drugs was designed in this research. The input and output of the strategy network are initially clarified. The strategy network captured by the drug retrieval robot performs mapping from the image of the captured drug to the position and posture information captured by the robot. Therefore, the input of the strategy network can be regarded as the candidate region image containing the position, posture, and shape information of the drug after drug detection (i.e., the state of the Markov decision process). The pose of the robot during grasping is expressed as g= (x,y,z,w,h,θ). In terms of the selected adsorption type end effector in the experiment, w and h represent the width and length of the medicine box, respectively, and θ=90°=π/2 is the angle in the w and h directions. The output is not the grasping pose of the robot but the pixel coordinates (u, v) corresponding to the grasping position of the robot. Rather, the position of the grasping point is calculated via the depth value and three-dimensional coordinate conversion as follows: The abovementioned parameters are all camera-related parameters. After the position of the pixel point in the camera coordinate system is obtained, the position of the pixel point in the robot coordinate system is generated via a homogeneous transformation of the camera coordinates and the dispensing robot coordinate system. The position obtained in the robot coordinate system mentioned above is a theoretical value. Due to error factors such as lens distortion in the vision system, the calibration method from reference [23] was selected during camera calibration, which to some extent reduced the existence of errors. Assume that the image size used is 256×256 and that u and v are within the interval [0,256]. During network construction, a simple mapping relationship u׳ =u×128 + 128 is used to map the value space of u and v to [−1,1]. u׳ represents the actual pixel coordinate. The analysis proves that the output of the strategy network is the variable (nx,ny,nz,nw,nh,nθ) at [−1,1]. According to the description of the capture actions for reference matrix boxes [24], each sampling point in the feature map is a captured reference rectangular box set at a certain scale and aspect ratio in the original image. The position of the grab rectangular box can be determined by the offset and the grab reference rectangular box. (nx,ny,nz,nw,nh,nθ) represents the geometric coordinate offset of the predicted grab rectangle obtained via regression from each rectangular box relative to the reference. Similarly, for each sampling point, corresponding to an n reference grabbing rectangular box, the output predicted grabbing rectangular box’s offset parameter dimension is 3n. Therefore, the output layer is equivalent to two 1 × 1 convolution kernels, which convolve the transition layer in terms of effect. The self-learning DDPG grasping strategy network model used to achieve an end-to-end output is shown in Fig 4. Network structure diagram of the grasping strategy of the medicine-taking robot. The size of the candidate capture area image is fixed at 256×256, which serves as the input for neural network training. The first few layers use a CNN for feature extraction, and the convolutional kernel size selected by the convolutional layer is 1*1. After a series of convolution operations, a feature map is obtained. The maximum pooling layer is used to reduce the image dimension after the main image features are extracted. The convolutional kernel selection of the maximum pooling layer is 2 × 2. The fully connected layer is subsequently used to obtain the intermediate transition layer. In the intermediate transition layer, in which the image feature and the corresponding variable outputs are considered, the parts for feature extraction and capture detection of the entire network, as well as the geometric parameters of the predicted capture rectangular box, are used to calculate the robot’s actions. The activation functions of the convolutional layer and the first two hidden layers of the fully connected layer are given by ReLU functions. The u function, with the final output layer taken as the tanh function, controls the output variables of the strategy [−1,1] mentioned above. Mapping of the grasping action function of the medication retrieval robot The neural network output of the drug retrieval robot’s grasping strategy consists of six variables with a value range of [−1,1]. This strategy is represented by (nx,ny,nz,nw,nh,nθ), which corresponds to the position (x,y), depth value z, width w, length h, and angle θ of the robot’s end effector. Therefore, a mapping relationship between the pixel point area (umax, umin, vmax, vmin) of the target drug from (nx,ny,nz,nw,nh,nθ) to (x,y,z,w,h,θ) needs to be established, and the main task is to determine the relationship between (nx,ny) and (x,y). The (nx, ny) output of the strategy network represents the relative position in the robot’s workspace. If the robot’s actions are limited to the candidate grasping area, then only (nx, ny) needs to be converted to the position within the relative candidate area. A coordinate system ouv is established with the center of the border of the candidate grasping area as the origin, and (nx, ny) is mapped to the pixel positions in the image according to the formula(5)-(6) (Fig 5). This scheme restricts the robot’s grasping position within the border of the candidate grasping area. The grasping posture of the drug-taking robot needs to be determined on the basis of the characteristics of different drugs. The transformation of the grasping posture output from the neural network to the drug-taking robot is computed as follows: Schematic of the coordinate system of the robot-grasping action. The method mentioned above maps the grasping point position to (x,y). Given that z is calculated using binocular images and that w, h, and θ are three parameters obtained on the basis of the characteristics of the medicine box, the final calculated grasping point position is (x,y,z,w,h,θ). In this research, in the calculation of the grasping pose by using an adsorption-type end effector, the optimal grasping rectangular box is the output in both the length and width directions with respect to the characteristics of the medicine box. The angle between the two directions is 90°. The intersection coverage area of the two directions is calculated to obtain the final adsorption pose. On the basis of the abovementioned mapping action, the position coordinates that the robot’s end effector needs to reach can be determined. During the actual grasping process, these position coordinates are sent to the end effector to execute grasping. Optimization of the action value function and exploration strategy design The action value function (Q function) represents the expected reward that a robot can receive when performing a certain action in the current state after reaching the termination state. The Q function is an important evaluation indicator in the Markov decision process and an imperative for judging the quality of a strategy. The strategy gradient method involves the weight coefficient identified before the gradient of the strategy function; the result determines the update direction of the strategy function. The Q function is represented by a neural network called the Q network. According to the basic principles of DRL, the input of the Q network should include the current state and the actions of the robot. The inputs of the Q network for the grasping task are the candidate grasping region image and (x,y,z,h,θ). The structure of the Q network is shown in Fig 6. Schematic of the Q network structure. The features extracted by the Q network are the same as those extracted by the policy network. The Q network adopts the same convolutional network as the policy network and shares the parameters of the convolutional network with the policy network during training. The candidate captured region images are not on the same order of magnitude as (x,y,z,w,h,θ), and their differences are relatively large. In this research, the convolutional layers of the CNN are used to extract image feature information and action values, which are then processed by a fully connected layer with 256 neurons. The data from each part are merged into the same fully connected layer. Here, a fully connected layer with 128 neurons and an output layer with 1 neuron are used. The DDPG is an offline DRL method that optimizes the Q network. The optimization objective function is given by: The Q network serves as both a part of the objective function and an update target; this configuration greatly increases the instability during the model training process. The Q value may also be overestimated, resulting from overfitting and causing the strategy network to learn incorrect information, which affects the final performance of the system. Thus, the Bellman equation of the Q function is used by the T-D algorithm [25] to establish a time difference. This difference is represented by an objective function, which can solve the problem of overestimation in DDPG and limit overfitting of the Q network. In the T-D algorithm, Q-learning [26] is the optimal algorithm used to solve offline learning. When a target network is added, the dual Q-learning algorithm [27] can be used to solve the instability problem in the training process. When the Q network is optimized, the dual Q-learning algorithm can use the target network to calculate the objective function. The network structure is shown in Fig 7. Double Q-learning network structure diagram. The task of the Actor is to find A and maximize the output Q: The target networks Q′ and µ′ for the Q network and policy network, respectively, are set according to Eq. (7). The objective function is calculated as follows: The target network of the dual Q-learning algorithm directly copies the various parameters of the original network; however, the original network needs to be updated every step during training. Typically, the target network is updated every step, with each update directly copying the parameters from the original network. As the target is updated every step, a delay error occurs when a corresponding update is performed in the Q network. The delay error generated in each step of the 0-N step increases as the number of steps increases. The target network avoids implementing updates every N steps to ensure a uniform delay error; that is, the updates are conducted every step. Compared with the original network, the update of the target network lags behind. The specific update formula is expressed as follows: θQ, θQ′,θμ, θμ′ are the Q network parameter, the Q function target network parameter, the robot strategy network parameter, and the robot strategy target network parameter, respectively. The learning stage of reinforcement learning corresponds to the autonomous exploration stage of the medication-dispensing robot. This robot explores all workspaces as much as possible and learns the advantages and disadvantages of actions on the basis of the reward function evaluation. As deterministic strategies are selected in this research, the drug retrieval robot repeatedly performs the same drug grasping action in the same state after initializing the strategy parameters and does not have exploratory ability. Random Gaussian white noise ζt~N(0,σ2) with a mean of 0 and a variance of σ2 is added to the action space for disturbance to enhance the exploration capability of the medication retrieval robot. where the variance σ2 controls the exploration space of the medication retrieval robot, θt=θ~−(θ~−θ) is the strategy network parameter after the addition of noise, θ~ is the parameter after disturbance of the strategy network, and θ is the parameter before disturbance. The Gaussian noise mean is 0. Thus, the best grasping action and actions far from the best grasping action, which is influenced by variance, have high probabilities of being selected. The strategy after the disturbance is denoted by μ~. The variance, which can be difficult to adjust manually under the influence of noise, must also be considered. Therefore, an adaptive adjustment method is adopted to measure the distance between strategy μ~ corresponding to parameter θt after the addition of noise and the original strategy µ as a measure of the scale of noise exploration control. In statistics, under the assumption of normal distribution, about 99.7% of the data will fall within the range of ±3σ. The standard deviation reflects the degree of dispersion between the data points and the mean. In the robot grasping space, the direction and range of the spatial set are determined by the standard deviation.The threshold δϵR is set as the standard deviation. A distance metric that is less than the threshold indicates insufficient disturbance, which increases the variance. The variance is decreased to obtain the updated formula for variance. where α=1.01 is the influencing factor. The structural diagram of the autonomous learning DDPG grasping algorithm for the drug-dispensing robot is shown in Fig 8, where f represents the mapping of actions and poses. The candidate grasping areas and policy network outputs are fused to ensure grasping actions by the robot in the candidate grasping areas. The attributes representing the robot’s state before executing the action, grasping actions, exploratory learning, and reward values are stored in the experience pool. Once the data in the experience pool reach a certain amount, the Q network can be trained from the experience pool according to uniformly sampled data. After training the Q network, the deterministic gradient method is used to update the robot human control strategy, and this process is repeated until convergence. Block diagram of the self-learning DDPG grabbing algorithm. The pseudocode for the self-learning DDPG algorithm process is shown in Table 1. Episode represents the number of training steps for reinforcement learning, and T is the duration sequence of the strategy in each scenario. The duration sequence captured by the medication retrieval robot is set to 1. During initialization, the Xavier initialization method [28] is selected, and the Adam optimizer is used by the neural network [29]. The Adam algorithm applies default hyperparameters, namely, a learning rate α of 0.001, decay rates of β1=0.9 and β2=0.999, and a setting of ε=10−8, to maintain numerical stability. 10.1371/journal.pone.0335675.t001Table 1Self-learning DDPG grabbing pseudocode for medicine-taking robots on the basis of detection constraints.Algorithm: Self-learning DDPG pseudocode grasping for medication retrieval robot based on detection constraintsInitialization: Q network Q(s,a|θQ) and grasping strategy network μ(s,a|θμ) parameters θQ and θμ, target networks Q′ and μ′:θQ′←θQ,θμ′←θμ, noise distribution standard deviation σ, experience pool size R1: for episode = 1, M do2: Get current status s3: for t = 1, T do4: Obtain actions through the current grasping strategy and exploring noise: at=μ(st|θμ)+ζt5: Execute action at to earn reward value rt and obtain the next state st+16: Store tuple (st,at,st+1,rt,ζt) in the experience pool7: Calculate metric distance d(μ,μ~), variance σ, and strategy network parameter θt8: Sample N tuples (st,at,st+1,rt,ζt) from the experience pool through uniform sampling9: Calculate yi=ri+γQ′(si+1,μ′(si+1|θμ′\rightleft|θQ′)10: Using Adam to optimize Q network parameters and minimize loss values: L=1N∑\nolimitsi(yi−Q(si,ai|θQ))211: Using deterministic policy gradient method to update policy network parameters:∇θμJ≈1N∑i∇aQ(s,a|θQ)|s=si,a=μ(si)∇θμμ(s|θμ\rightleft|s12: Update target network parameters:θQ′←τθQ+(1−τ)θQ′θμ′←τθμ+(1−τ)θμ′13: end for14: end for Self-learning DDPG grabbing pseudocode for medicine-taking robots on the basis of detection constraints. Experiments and results The self-learning DDPG grasping algorithm includes two parts. First, the calculated grasping point pose is output via the strategy network, which is known as grasping point detection and serves as the input of the Q network. Second, the robot’s autonomous grasping ability is trained via reinforcement learning. The experiments performed in this study were validated from two aspects. First, the accuracy of the strategy network’s grasping point detection was verified on a public dataset and applied to actual drug-grasping point detection. Second, the success rate of the robot’s autonomous learning and grasping was simulated and calculated for the autonomous learning network. Verification and analysis of the grasping point detection results (1) Standard dataset capture point detection verification. The grasping dataset [30] of Cornell University was selected for the experiment, from which the accuracy of the output grasping point pose of the strategy network model was verified. The Cornell dataset includes 1035 images, which cover 280 different everyday objects. Fig 9 shows a section of this dataset. In the captured dataset, the precaptured poses are represented by the captured rectangular boxes labeled in the images. The images are used as network inputs, whereas the labeled captured rectangular boxes are used as outputs. The captured rectangular boxes in this section are output only in a single direction. After training, the proposed strategy network can successfully detect the captured rectangular boxes suitable for capturing the target object from the images. Cornell crawled dataset. Point detection verification is performed on some images in the Cornell dataset via the proposed strategy network. The results of the grasping detection are shown in Fig 10. The first row represents the image input of the model, and the second and third rows represent the outputs of the model. The second row is represented by a schematic of the reference rectangular box captured at each position, with all available positions marked to select the most suitable position for capture. The network can find suitable positions for grasping the target object from the image. For some target objects with obvious handles, the network can output predicted grasping rectangles near the handles. The third row of images displays the highest scoring predicted grasping rectangles, which can sufficiently reflect the pose suitable for grasping the target. The results of the grasping detection. Table 2 lists the detection accuracy results of the mainstream methods and the proposed model on the Cornell dataset. The final performance metrics are calculated as follows, select 50 objects in the dataset, detect each object 10 times, and calculate the average grasping detection accuracy ± standard deviation. The grasping detection accuracy of the proposed method is high, indicating the advantages of self-learning strategy networks. Attribute gains to candidate region extraction (reduced search space) and adaptive DDPG exploration. Although the running speed of the proposed model is slightly lower than that of Redomon, its accuracy is improved by 15%. Compared with other methods, the proposed model performs better in terms of both capture detection accuracy and running speed, the standard deviation is also the smallest. 10.1371/journal.pone.0335675.t002Table 2Grab detection accuracy and running time of different methods.MethodDetection accuracyTimesLuca [31]72.9% ± 4.715.7sLenz73.9% ± 3.914.5sRedomon [32]86% ± 3.60.86sKim [33]76.3% ± 4.517.6sOurs91.7% ± 2.30.9s Grab detection accuracy and running time of different methods. (2) Actual drug-grasping point detection results. The accuracy of the strategy network model for drug-grasping point detection calculation was evaluated by subjecting actual drugs to grasping point calculation verification, such as single-drug-grasping point detection and multidrug-grasping point detection. The images of the candidate grasping region are used as the test images for the proposed network model. All possible grasping reference rectangular boxes were considered. The results of grasping point detection are shown in Fig 11. The image input is represented by the first-row model, whereas the output is represented by the second and third rows. The second row corresponds to a schematic of all graspable position reference rectangles. The third line of the image shows the final predicted position of the captured rectangular box. The characteristics of the medicine box (i.e., the grasping rectangular box for optimal prediction) suggest the need to determine the optimal positions in both the horizontal and vertical directions. The proposed method can find a suitable position for grasping the medicine from the image, and the predicted grasping rectangular box can sufficiently reflect the posture suitable for grasping the medicine. For the grasping posture of the adsorption end effector, the center coordinates of the overlapping area in both directions correspond to the optimal grasping detection points. During testing, the network utilizes GPU computing, with a detection rate of up to 70 frames per second, which can meet the real-time grasping needs of robots. The results of grasping point detection. Table 3 lists the statistical results of the accuracy of the investigated mainstream methods and the network model strategy proposed in this study for detecting the grasping points of a single drug. The final performance metrics are calculated as follows, select three drugs, twenty calculations were performed for each drug, and determine the average capture detection accuracy ± standard deviation. Compared with the other four methods, the proposed network model has the highest accuracy in detecting drug-grasping points. Although the running speed of the proposed algorithm approximates that of Redomon’s method, it is higher than those of the other three algorithms, The standard deviation is also the smallest. The proposed algorithm demonstrates the good performance of the network model. 10.1371/journal.pone.0335675.t003Table 3Grab detection accuracy and running time of different methods.MethodDrug 1Drug 2Drug 3Detection accuracyTimesDetection accuracyTimesDetection accuracyTimesLuca69.3% ± 5.218.1s72.6% ± 5.115.6s74.3% ± 5.117.9sLenz72.1% ± 3.914.5s73.1% ± 3.513.5s74.7% ± 3.715.2sRedomon82.9% ± 2.81.2s83% ± 2.61.3s83.9% ± 2.61.5sKim71.2% ± 3.715.7s72.6% ± 3.716.9s71.5% ± 3.718.3sOurs90.6% ± 0.01.1s92.8% ± 2.11.3s92.3% ± 0.01.3s Grab detection accuracy and running time of different methods. Grasping point detection calculations were performed in the presence of multiple drugs to further verify the generalizability of the model. The visualization effect is shown in Fig 12. The first image shows the input image, The second image shows all the detectable grasping poses, and The third image shows the optimal grasping pose. The recognition results reveal that the proposed algorithm can identify the position information of each target drug in the image and accurately calculate the grasping pose of each drug. The proposed algorithm has also shown excellent performance in calculating the grasping points of multiple target drugs. The visualization effect. Table 4 lists several methods and the proposed model and their ability to calculate the accuracy and running time of grasping point detection for multiple target drugs. Twenty grasping detections for each of the four drugs were observed. Calculate the average grasping detection accuracy ± standard deviation. The data represent the statistical results of multiple grasping point detection calculations. Compared with the other four methods, the proposed method has the highest accuracy in detecting the grasping points of drugs. For drugs 1 and 2, the detection accuracies exceeded about 80%. Drug 4 has a relatively small medicine box, and the detection accuracy reached 75%, the standard deviation is also the smallest. In terms of running time, the proposed approach has the shortest running time among several methods for testing multiple drugs. 10.1371/journal.pone.0335675.t004Table 4Grab detection accuracy and running time of different methods.MethodDrug 1Drug 2Drug 3Drug 4Detection accuracyTimesDetection accuracyTimesDetection accuracyTimesDetection accuracyTimesLuca67.3% ± 4.215.3s58.6% ± 4.416.3s61.7% ± 4.315.7s53.4% ± 6.416sLenz72.4% ± 3.916.2s62.4% ± 3.916.8s64.1% ± 3.815.9s58.9% ± 6.116.8sRedomon80.7% ± 3.51.5s73.7% ± 4.32.9s76.8% ± 3.51.7s68.2% ± 5,21.7sKim72.2% ± 3.811.5s64.2% ± 4.312.3s68.6% ± 3.811.8s60.1% ± 5.812.7sOurs87.9% ± 2.11.3s79.8% ± 2.82s82.2% ± 2.71.5s76.9% ± 2.81.3s Grab detection accuracy and running time of different methods. Self-directed learning to capture simulation results A simple robot model was established in a simulation environment to verify the superiority of the self-learning DDPG algorithm over the traditional DDPG algorithms in terms of learning efficiency. The abovementioned methods were used to learn how to grasp rectangular prisms. Learning to grasp rectangular prisms is necessary because the drugs captured in this study were designed mainly for boxed drugs, and the medicine box could be analogized to a rectangular prism (Fig 13). During the grasping simulation, the position and posture of the rectangular prism were constantly changing. Thus, the output of the strategy network was considered, and the Q network was used to evaluate the quality of the strategy network model. Force feedback during grasping was used to determine whether an object would be successfully grasped. In this scheme, if the feedback force exceeds the threshold, then the robot has successfully grasped, and a reward value of 1 is given. If the feedback force is less than the threshold, then the robot has failed to grasp, and a penalty value of −1 is given. Schematic of the grab simulation. During training, the traditional DDPG algorithm、Twin Delay DDPG(TD3)、Soft Actor-Critic(SAC)and the self-learning DDPG algorithm performed similarly, but they differed in terms of the sampling method. The four algorithms shared the same grasping strategy network, action value function, and exploration strategy. They were also trained at 20000 steps each, and their performance metrics were evaluated using the grasping success rate indicator. The final experimental results are shown in Fig 14. Fig 15 shows the results of several algorithms within a 95% confidence interval. Grab success rates of the two algorithms under the same number of training steps. The performance of four algorithms with 95% confidence intervals. The experimental results show that the model basically converges after 20,000 training steps. Compared with the convergence speeds of the four algorithms, the proposed algorithm outperforms the SAC algorithm, while the DDPG algorithm has the slowest convergence speed. In terms of grasping success rate, the proposed algorithm also achieves a higher success rate than the other three algorithms. This simulation experiment verifies that the self-learning DDPG algorithm has a higher learning speed than the other algorithms do in terms of grasping tasks, thereby achieving the goal of improving the learning speed of the DDPG algorithm via self-learning. Actual drug retrieval To verify the effectiveness of this method in real-world robotic grasping, a set of experimental tasks for a medicine-fetching robot to grasp medicines in real-world scenarios was established. It utilized a vision system in conjunction with the medicine-fetching robot. During the experiment, the drug is placed beneath the visual system, which allows the camera to capture images of the current experimental scene in real-time. The drug detection algorithm efficiently extracts drug information from the image data stream. Then, it locates the captured drug in the current scene. Next, it generates a candidate grasping area where the drug is located. Finally, it utilizes the self-learning DDPG end-to-end neural network to optimize the grasping pose of the drug within the candidate area in real time. Based on the selected end effector, vertical grasping is adopted, where the end effector is placed above the drug with the z-axis direction perpendicular to the operating plane. The grasping position is determined by the overlapping area of the rectangular boxes representing the drug pose in two directions, which controls the robot to reach the pre-grasping pose for grasping. After executing the end stop action, the robot is controlled to move to the designated placement position, complete the drug grasping operation, and wait to return to the initial pose. The motion process of the robot from preparation to grasping and finally placing the medicine in Fig 16. The process of robot positioning and grabbing medicine. To compare the applicability of the positioning and grasping algorithm presented in this study for medicine-fetching robots, this chapter contrasts the proposed method with two methods proposed by Guo and Johns [34] for guiding and grasping medicine-fetching robots. The three methods were tested on single and multiple medicine targets separately. In the single-target medicine grasping experiment, a target medicine was placed on the medicine conveyor belt, and the three methods were used to repeatedly grasp the medicine for 30 times. The experimental statistics for the detection accuracy and grasping success rate± standard deviation. of the medicine are shown in Table 5, where the data are the average values calculated from 30 trials. 10.1371/journal.pone.0335675.t005Table 5Different methods capture statistical results for four single-target drugs.JohnsGuoOursDetection accuracyGrasping accuracyDetection accuracyGrasping accuracyDetection accuracyGrasping accuracyDrug 186.7% ± 2.373.3% ± 3.483.3% ± 3.280% ± 3.296.7% ± 0.090% ± 1.2Drug 276.7% ± 6.466.7% ± 5.676.7% ± 5.970% ± 5.993.3% ± 0.083.3% ± 1.2Drug 380% ± 2.373.3% ± 2.380% ± 3.576.7% ± 4.593.3% ± 0.086.7% ± 1.2Drug 466.7% ± 5.663.3% ± 5.673.3% ± 5.666.7% ± 5.990% ± 1.683.3% ± 4.0 Different methods capture statistical results for four single-target drugs. The robot was instructed to grasp multiple drugs instead of a single one to verify the applicability of the detection algorithm. The robot was repeatedly used to grasp drugs 30 times, and the statistical results of the detection accuracy and grasping success rate± standard deviation. are presented in Table 6. 10.1371/journal.pone.0335675.t006Table 6Statistical results of multi-target drug detection and capture by different methods.JohnsGuoOursDetection accuracyGrasping accuracyDetection accuracyGrasping accuracyDetection accuracyGrasping accuracyDrug 180% ± 4.570% ± 5.683.3% ± 3.673.3% ± 4,790% ± 0.083.3% ± 4.2Drug 266.7% ± 6.760% ± 6.770% ± 5.860% ± 6.586.7% ± 1.280% ± 2.8Drug 376.7% ± 7.366.7% ± 6.576.7% ± 5.870% ± 5.983.3% ± 2.180% ± 2.8Drug 466.7% ± 6.756.7% ± 9.870% ± 5.860% ± 6.580% ± 2.173.3% ± 4.0 Statistical results of multi-target drug detection and capture by different methods. Finally, the drugs grasped by the robot were transformed into two identical targets to verify the applicability of the detection algorithm. The drugs were grasped 30 times, and the statistical results of the detection accuracy and grasping success rate± standard deviation. of the drugs are presented in Table 7. 10.1371/journal.pone.0335675.t007Table 7Statistical results of two identical drug detection and capture by different methods.JohnsGuoOursDetection accuracyGrasping accuracyDetection accuracyGrasping accuracyDetection accuracyGrasping accuracyDrug 183.3% ± 4.570% ± 3.780% ± 3.973.3% ± 6.196.7% ± 0.090% ± 0.0Drug 286.7% ± 3.873.3% ± 3.783.3% ± 4.770% ± 6.193.3% ± 1.890% ± 0.0 Statistical results of two identical drug detection and capture by different methods. The experimental data presented in Tables 5-3 to 5-5 demonstrate that the monitoring accuracy and grasping success rate of the robot grasping algorithm proposed in this study are relatively high. Therefore, it can handle the actual operation of medicine retrieval and dispensing in automated pharmacies. In addition, it exhibits applicability in recognizing and grasping single categories, single drugs, multiple categories, multiple drugs within the same category, and drugs in arbitrary poses. Conclusions A method for point detection via DDPG autonomous learning was proposed on the basis of penalty constraints and DRL. First, drawing on the powerful image feature extraction ability of deep CNNs, a strategy network for the drug retrieval robot to grasp drugs was designed, and the calculated grasping point pose was output via the strategy network. Second, reinforcement learning was combined with autonomous learning, allowing the robot’s autonomous grasping ability to be trained. The experiment verified the accuracy of the strategy network’s point detection from two aspects: as applied to a public dataset and as applied to actual drug point detection. The accuracy and time of point detection of the proposed method were better than those of the other algorithms. Finally, simulation calculations were conducted on the autonomous learning network to verify the success rate of robot autonomous learning and grasping. Compared with the traditional DDPG algorithm, the algorithm proposed in this study has a higher learning speed and can achieve the goal of improving the learning speed of the DDPG algorithm via autonomous learning. Supporting information Relevant data underlying the findings described in Fig 14. (XLSX) Relevant codes underlying the findings described in manuscript. (TXT) 25 Jul 2025 Dear Dr. Zhang, Thank you for submitting your manuscript to PLOS ONE. After careful consideration, we feel that it has merit but does not fully meet PLOS ONE’s publication criteria as it currently stands. Therefore, we invite you to submit a revised version of the manuscript that addresses the points raised during the review process. ============================== ACADEMIC EDITOR: The manuscript presents many major concerns as highlighted by both reviewers. However, I see merits on the work. Please, address all the concerns rised by the Experts and provide an updated version of the manuscript. ============================== Please submit your revised manuscript by Sep 08 2025 11:59PM. If you will need more time than this to complete your revisions, please reply to this message or contact the journal office at plosone@plos.org . When you're ready to submit your revision, log on to https://www.editorialmanager.com/pone/ and select the 'Submissions Needing Revision' folder to locate your manuscript file. A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'.A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'.An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A rebuttal letter that responds to each point raised by the academic editor and reviewer(s). You should upload this letter as a separate file labeled 'Response to Reviewers'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. A marked-up copy of your manuscript that highlights changes made to the original version. You should upload this as a separate file labeled 'Revised Manuscript with Track Changes'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. An unmarked version of your revised paper without tracked changes. You should upload this as a separate file labeled 'Manuscript'. If you would like to make changes to your financial disclosure, please include your updated statement in your cover letter. Guidelines for resubmitting your figure files are available below the reviewer comments at the end of this letter. If applicable, we recommend that you deposit your laboratory protocols in protocols.io to enhance the reproducibility of your results. Protocols.io assigns your protocol its own identifier (DOI) so that it can be cited independently in the future. For instructions see: https://journals.plos.org/plosone/s/submission-guidelines#loc-laboratory-protocols . Additionally, PLOS ONE offers an option for publishing peer-reviewed Lab Protocol articles, which describe protocols hosted on protocols.io. Read more information on sharing protocols at https://plos.org/protocols?utm_medium=editorial-email&utm_source=authorletters&utm_campaign=protocols . We look forward to receiving your revised manuscript. Kind regards, Andrea Tigrini, Ph.D. Academic Editor PLOS ONE Journal Requirements: When submitting your revision, we need you to address these additional requirements. 1. Please ensure that your manuscript meets PLOS ONE's style requirements, including those for file naming. The PLOS ONE style templates can be found at https://journals.plos.org/plosone/s/file?id=wjVg/PLOSOne_formatting_sample_main_body.pdf and https://journals.plos.org/plosone/s/file?id=ba62/PLOSOne_formatting_sample_title_authors_affiliations.pdf . 2. Please note that PLOS ONE has specific guidelines on code sharing for submissions in which author-generated code underpins the findings in the manuscript. In these cases, we expect all author-generated code to be made available without restrictions upon publication of the work. Please review our guidelines at https://journals.plos.org/plosone/s/materials-and-software-sharing#loc-sharing-code and ensure that your code is shared in a way that follows best practice and facilitates reproducibility and reuse. 3. We note that your Data Availability Statement is currently as follows: [All relevant data are within the manuscript and its Supporting Information files.] Please confirm at this time whether or not your submission contains all raw data required to replicate the results of your study. Authors must share the “minimal data set” for their submission. PLOS defines the minimal data set to consist of the data required to replicate all study findings reported in the article, as well as related metadata and methods (https://journals.plos.org/plosone/s/data-availability#loc-minimal-data-set-definition). For example, authors should submit the following data: - The values behind the means, standard deviations and other measures reported; - The values used to build graphs; - The points extracted from images for analysis. Authors do not need to submit their entire data set if only a portion of the data was used in the reported study. If your submission does not contain these data, please either upload them as Supporting Information files or deposit them to a stable, public repository and provide us with the relevant URLs, DOIs, or accession numbers. For a list of recommended repositories, please see https://journals.plos.org/plosone/s/recommended-repositories. If there are ethical or legal restrictions on sharing a de-identified data set, please explain them in detail (e.g., data contain potentially sensitive information, data are owned by a third-party organization, etc.) and who has imposed them (e.g., an ethics committee). Please also provide contact information for a data access committee, ethics committee, or other institutional body to which data requests may be sent. If data are owned by a third party, please indicate how others may request data access. 4. We note that the grant information you provided in the ‘Funding Information’ and ‘Financial Disclosure’ sections do not match. When you resubmit, please ensure that you provide the correct grant numbers for the awards you received for your study in the ‘Funding Information’ section. 5. Thank you for stating the following in the Acknowledgments Section of your manuscript: “This work was supported by the Youth Project of Shanxi Province Basic Research Program (Free Exploration Category, 202303021222299).” We note that you have provided additional information within the Acknowledgements Section that is not currently declared in your Funding Statement. Please note that funding information should not appear in the Acknowledgments section or other areas of your manuscript. We will only publish funding information present in the Funding Statement section of the online submission form. Please remove any funding-related text from the manuscript and let us know how you would like to update your Funding Statement. Currently, your Funding Statement reads as follows: “The author(s) received no specific funding for this work.” Please include your amended statements within your cover letter; we will change the online submission form on your behalf. 6. If the reviewer comments include a recommendation to cite specific previously published works, please review and evaluate these publications to determine whether they are relevant and should be cited. There is no requirement to cite these works unless the editor has indicated otherwise. Additional Editor Comments (if provided): The manuscript presents many major concerns as highlighted by both reviewers. However, I see merits on the work. Please, address all the concerns rised by the Experts and provide an updated version of the manuscript. [Note: HTML markup is below. Please do not edit.] Reviewers' comments: Reviewer's Responses to Questions Comments to the Author 1. Is the manuscript technically sound, and do the data support the conclusions? Reviewer #1: Partly Reviewer #2: No ********** 2. Has the statistical analysis been performed appropriately and rigorously? -->?> Reviewer #1: No Reviewer #2: No ********** 3. Have the authors made all data underlying the findings in their manuscript fully available??> The PLOS Data policy Reviewer #1: Yes Reviewer #2: No ********** 4. Is the manuscript presented in an intelligible fashion and written in standard English??> Reviewer #1: Yes Reviewer #2: No ********** Reviewer #1: The manuscript proposes a DDPG-based autonomous learning algorithm for drug retrieval robots, it addresses challenges in grasping accuracy and real-time performance. While the methodology is interesting, the manuscript requires significant improvements in clarity, technical rigor, and validation to meet publication standards. Below are detailed concern and recommendations. 1. The "detection constraints" (central to the title) are never formally defined. It is not clear how boundaries are analysed (Section 3.1). The constraints must be mathematically defined and how much candidate region extraction boost performance should be evaluated and justified. 2. Equation (8) assumes a pinhole camera model but omits lens distortion correction, critical for real-world accuracy. Please take this into account. 3. The adaptive noise exploration (Eq. 17) lacks justification for threshold δ and decay factor α. Please provide these justifications. 4. "The proposed method has more accurate grasping poses" in the Abstract is not statistically tested. It should be added. 5. Latency (0.9s) is claimed but not benchmarked against robot control cycles. Please compare latency to robot actuator response times (e.g., typical 10–100ms). 6. The autonomous learning component is compared only to vanilla DDPG, not state-of-the-art alternatives (e.g., SAC - Soft Actor-Critic, TD3 - Twin Delayed Deep Deterministic Policy Gradient). Please add alternative comparisons. 7. In the introduction challenging grasping problems such as those showed in the paper “Automating the hand layup process: On the removal of protective films with collaborative robots” https://doi.org/10.1016/j.rcim.2024.102899, should be mentioned. 8. The manuscript reveal a limited real-world testing, experiments use public datasets (Cornell) and simple simulations (rectangular prism). Real robot trials are absent. Test on a physical robot with varied drug geometries (vials, blister packs) is recommended. 9. The DDPG training in Section 4.3.2 uses a simplistic rectangular prism, ignoring real drug variability. Please validate autonomous grasping in a physics simulator with randomized drug poses. 10. Superior accuracy (92.1% vs. 86.4% in Table 2) is presented without technical explanation. Is this due to boundary analysis or DDPG? Please clarify. Please attribute gains to candidate region extraction (reduced search space) and adaptive DDPG exploration. 11. Statistical Reporting in Fig. 12 seems to be inadequate; the success rates (Fig. 12) lack variance measures. Please add confidence intervals to Fig. 12; report mean ± std. dev. in tables. 12. Results are disconnected from real-world pharmacy demands (e.g., FDA standards for medication retrieval accuracy) Please discuss applicability in hospital settings, citing safety margins (e.g., Reviewer #2: The proposed work aimed at providing a method for optimizing autonomous robot grasping for drug retrieval using reinforcement learning, specifically deep deterministic policy gradient (DDPG) algorithm. I have some major concerns that mainly regard the writing and the methodological rigor. My major doubt regards the novelty of the proposed approach: it is not clear if a novel approach was proposed or if an already known approach was applied to the specific problem of drug retrieval. This should have been better clarified throughout the text. Moreover, the description of the performed experiments is lacking, thus hampering a thorough evaluation of the methodological power. Point by point comments are provided in the following. - Abstract should report some numerical results. - Aim of the study should be clearly stated and should be more specific about the novelty. - The Related Work section is not informative and needs to be rewritten. Related Works should report in brief the relevant literature in the field of the deep reinforcement learning strategies and the state of the art and latest relevant methods for the field of drug retrieval with robot grasping. Replicating theoretical formulas already consolidated makes the manuscript very cumbersome. - In the references section there are no references regarding automatic drug retrieval with robot grasping. This raises concerns about whether the research question is grounded in current developments or unmet needs in the field. Recent work in pharmacy automation and robotic manipulation should be cited to better contextualize the relevance and novelty of the contribution. - The Methods (section 3), as it is written, do not allow to understand what part of the proposed method is new with respect to the existing literature. - In section 3.2 it is stated “ nx ny is mapped to the pixel positions in the image according to the formula (Fig. 3).”, but no equation is cited and a figure is mentioned instead. Which “formula” is this sentence referring to? - In the results section 4.3.1, how was the detection accuracy computed? What does “The images of the candidate grasping region are used as the test images for the proposed network model” mean? How was the dataset of the “(2) Actual drug-grasping point detection results” experiment obtained? Materials used and performed experiments must be carefully explained. - The basis for the statement “The proposed method can find a suitable position for grasping the medicine from the image, and the predicted grasping rectangular box can sufficiently reflect the posture suitable for grasping the medicine.” is unclear. Is this claim supported by quantitative evaluation? The red lines in Figure 9 are difficult to be interpreted. - It is not specified why the algorithms used for comparisons where selected: where those algorithms the gold standards? Moreover, are those algorithms/approaches open source? It should be specified if such algorithms where implemented to carry on comparison experiments. - Formatting of references should be carefully assessed. Reference n 29 is wrongly formatted (i.e., first names are reported instead of last names). ********** what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy Reviewer #1: No Reviewer #2: No ********** [NOTE: If reviewer comments were submitted as an attachment file, they will be attached to this email and accessible via the submission site. Please log into your account, locate the manuscript record, and check for the action link "View Attachments". If this link does not appear, there are no attachment files.] While revising your submission, please upload your figure files to the Preflight Analysis and Conversion Engine (PACE) digital diagnostic tool, https://pacev2.apexcovantage.com/ . PACE helps ensure that figures meet PLOS requirements. To use PACE, you must first register as a user. Registration is free. Then, login and navigate to the UPLOAD tab, where you will find detailed instructions on how to use the tool. If you encounter any issues or have any questions when using PACE, please email PLOS at figures@plos.org . Please note that Supporting Information files do not need this step. 22 Sep 2025 List of Responses Dear Editors and Reviewers On behalf of my co-authors, we thank you very much for giving us an opportunity to revise our manuscript, we appreciate editor and reviewers very much for their positive and constructive comments and suggestions on our manuscript entitled “An algorithm for drug retrieval based on robot-grasping detection constraints and DDPG autonomous learning”. (ID: PONE-D-25-23921). Those comments are all valuable and very helpful for revising and improving our paper, and they are also having important guiding significance to us researches. We have studied those comments carefully and have made correction which are marked in red in the paper, hoping they will meet the standard. The main corrections and the responds to the reviewer’s comments are as flowing: Responds to the reviewer’s comments: Reviewer #1: 1. [Response to comment]: The "detection constraints" (central to the title) are never formally defined. It is not clear how boundaries are analysed (Section 3.1). The constraints must be mathematically defined and how much candidate region extraction boost performance should be evaluated and justified. [Responds]: We sincerely apologize that this part of the content was not detailed enough in the manuscript. In section ‘Candidate capture area generation’, the content of detection constraints and boundary analysis is described in detail, where the location of the drug can be determined through candidate region generation. 2. [Response to comment]: Equation (8) assumes a pinhole camera model but omits lens distortion correction, critical for real-world accuracy. Please take this into account. [Responds]: Considering the reviewer’s suggestion, Equation (8) is Equation (3), we have supplemented the camera calibration process by utilizing the method described in reference [23], which has reduced the error caused by distortion to a certain extent. 3. [Response to comment]: The adaptive noise exploration (Eq. 17) lacks justification for threshold δ and decay factor α. Please provide these justifications. [Responds]: Thank you for reviewer’s suggestion. We apologize for discovering a writing error α=1.01. In statistics, under the assumption of normal distribution, about 99.7% of the data will fall within the range of . The standard deviation reflects the degree of dispersion between the data points and the mean. In the robot grasping space, the direction and range of the spatial set are determined by the standard deviation, so the threshold is chosen as the standard deviation. In terms of implementation, different algorithms have different distance metrics d before and after disturbance. If the distance metric is less than the threshold, it indicates that the disturbance is not sufficient and the search space needs to be increased. On the contrary, if the distance metric is greater than the threshold, it indicates that the disturbance is too large and the search space needs to be reduced. 4. [Response to comment]: "The proposed method has more accurate grasping poses" in the Abstract is not statistically tested. It should be added. [Responds]: According to the reviewer’s suggestion, "The proposed method has more accurate grasping poses" in the Abstract. In the experimental process, the main focus was on statistical description of the capture detection accuracy in different datasets. We have changed it to a description using the capture detection accuracy in the abstract. 5. [Response to comment]: Latency (0.9s) is claimed but not benchmarked against robot control cycles. Please compare latency to robot actuator response times (e.g., typical 10–100ms). [Responds]: In the experimental part, latency (0.9s) refers to the time it takes for the visual system to capture images and output the grasping pose, excluding the average response time of the robot. 6. [Response to comment]: The autonomous learning component is compared only to vanilla DDPG, not state-of-the-art alternatives (e.g., SAC - Soft Actor-Critic, TD3 - Twin Delayed Deep Deterministic Policy Gradient). Please add alternative comparisons. [Responds]: We sincerely apologize that the content didn't describe this part in detail enough in the comparative experiment. In section ‘Self-directed learning to capture simulation results’, we included a comparison with SAC and TD3 algorithms. 7. [Response to comment]: In the introduction challenging grasping problems such as those showed in the paper “Automating the hand layup process: On the removal of protective films with collaborative robots” https://doi.org/10.1016/j.rcim.2024.102899, should be mentioned. [Responds]: Considering the reviewer’s suggestion, we mentioned the challenging grasping problems with this paper in the introduction, and marked in red. 8. [Response to comment]: The manuscript reveal a limited real-world testing, experiments use public datasets (Cornell) and simple simulations (rectangular prism). Real robot trials are absent. Test on a physical robot with varied drug geometries (vials, blister packs) is recommended. [Responds]: In the experimental part, we have added real robot trials, and the target object in the paper is boxed drugs, so the experiment mainly focuses on detecting and grasping boxed drugs. We will consider grasping other varied drug geometries (vials, blister packs) for future work. 9. [Response to comment]: The DDPG training in Section 4.3.2 uses a simplistic rectangular prism, ignoring real drug variability. Please validate autonomous grasping in a physics simulator with randomized drug poses. [Responds]: Due to the fact that our main application in automated pharmacies is boxed drugs, and we also use adsorption end effectors, in order to verify the effectiveness of candidate area boundary analysis and self-learning networks, we only trained rectangular prisms in Section ‘Self-directed learning to capture simulation results’. 10. [Response to comment]: Superior accuracy (92.1% vs. 86.4% in Table 2) is presented without technical explanation. Is this due to boundary analysis or DDPG? Please clarify. Please attribute gains to candidate region extraction (reduced search space) and adaptive DDPG exploration. [Responds]: Considering the reviewer’s suggestion, the detection accuracy in Table 2 is mainly due to the boundary analysis of candidate grasping areas, the input of images into the self-learning detection network, and the final output of the target's grasping detection accuracy. It has been marked in red in the manuscript. 11. [Response to comment]: Statistical Reporting in Fig. 12 seems to be inadequate; the success rates (Fig. 12) lack variance measures. Please add confidence intervals to Fig. 12; report mean ± std. dev. in tables. [Responds]: Thank you for reviewer’s suggestion. We apologize for not describing it enough in the manuscript. In Fig 14, we added confidence intervals and included mean ± std in the table. 12. [Response to comment]: Results are disconnected from real-world pharmacy demands (e.g., FDA standards for medication retrieval accuracy) Please discuss applicability in hospital settings, citing safety margins (e.g.). [Responds]: Thank you for reviewer’s suggestion. Currently, our method is in the testing stage in automated pharmacies and will be further improved based on your valuable suggestion. Reviewer #2: 1. [Response to comment]: Abstract should report some numerical results. [Responds]: Thank you for reviewer’s suggestion, we did not consider everything comprehensively in the manuscript, and have already included numerical results in the Abstract. 2. [Response to comment]: Aim of the study should be clearly stated and should be more specific about the novelty. [Responds]: Considering the reviewer’s suggestion, we have rewritten the aim of the study, clearly describing the quality of the proposed work, and marked in red in the manuscript. 3. [Response to comment]: The Related Work section is not informative and needs to be rewritten. Related Works should report in brief the relevant literature in the field of the deep reinforcement learning strategies and the state of the art and latest relevant methods for the field of drug retrieval with robot grasping. Replicating theoretical formulas already consolidated makes the manuscript very cumbersome. [Responds]: Considering the reviewer’s suggestion, we have written the Related Work section. Added the relevant paper in the field of the deep reinforcement learning strategies and the state of the art and latest relevant methods for the field of drug retrieval with robot grasping. 4.[ Response to comment]: In the references section there are no references regarding automatic drug retrieval with robot grasping. This raises concerns about whether the research question is grounded in current developments or unmet needs in the field. Recent work in pharmacy automation and robotic manipulation should be cited to better contextualize the relevance and novelty of the contribution. [Responds]: Thank you for reviewer’s suggestion. We have added new reference on the direction of robot grasping for automatic medication retrieval. 5. [Response to comment]: The Methods (section 3), as it is written, do not allow to understand what part of the proposed method is new with respect to the existing literature. [Responds]: Considering the reviewer’s suggestion, we have reiterated the novelty of the proposed method in section ‘Proposed method’. 6. [Response to comment]: In section 3.2 it is stated “nx ny is mapped to the pixel positions in the image according to the formula (Fig. 3).”, but no equation is cited and a figure is mentioned instead. Which “formula” is this sentence referring to? [Responds]: We are very sorry that we did not express this part in detail in the manuscript. Formulas refer to formulas 5 and 6. 7. [Response to comment]: In the results section 4.3.1, how was the detection accuracy computed? What does “The images of the candidate grasping region are used as the test images for the proposed network model” mean? How was the dataset of the “(2) Actual drug-grasping point detection results” experiment obtained? Materials used and performed experiments must be carefully explained. [Responds]: In the manuscript, we have detailed the deep topology, how the parameters are tuned and selected. They are not dependent on images. (1) In the results section ‘Verification and analysis of the grasping point detection results’, The detection accuracy is calculated using the following formula: (2) We sincerely apologize for the insufficient description. ‘The images of the candidate grasping region are used as the test images for the proposed network model’ refers to the result image of candidate grasping regions output in section ‘Candidate capture area generation’, which serves as input to the policy network, enabling targeted output of grasping poses. (3) The actual drug capture dataset is a self-built dataset, which contains 50 real target drugs, with 200 training images established for each drug. 8. [Response to comment]: The basis for the statement “The proposed method can find a suitable position for grasping the medicine from the image, and the predicted grasping rectangular box can sufficiently reflect the posture suitable for grasping the medicine.” is unclear. Is this claim supported by quantitative evaluation? The red lines in Figure 9 are difficult to be interpreted. [Responds]: In the experimental part, using both publicly available and self-built datasets for point detection and validation, the output rectangular box represents the appropriate location for grabbing drugs. The red line in Figure 9 (b) is a schematic diagram of all reference rectangular boxes that can be grabbed. Figure 9 (c) shows the final predicted grasping rectangle box, which is grasped at the center position using an adsorption type end effector. 9. [Response to comment]: It is not specified why the algorithms used for comparisons where selected: where those algorithms the gold standards? Moreover, are those algorithms/approaches open source? It should be specified if such algorithms where implemented to carry on comparison experiments. [Responds]: In the experimental part, the comparison algorithms are not gold standards, but they all have their advantages. [9] is the first application of deep learning networks for grasping point detection. [31] highlights the real-time performance of point detection based on deep learning for grasping. [32] mainly highlights the real-time performance of using neural networks for grasping point detection. [33] mainly highlights the real-time performance of grasping point detection. These methods are all open-source. 10. [Response to comment]: Formatting of references should be carefully assessed. Reference n 29 is wrongly formatted (i.e., first names are reported instead of last names). [Responds]: Thank you for reviewer’s suggestion. We have reconstructed, checked, and revised the reference format. Thank you for your constructive comments. We tried our best to improve the manuscript and made some changes in the manuscript, already marked in red. These changes will not influence the content and framework of the paper. We need to show our heartfelt appreciation for Editors/Reviewers’ efforts, and hope that the correction will meet with approval. Once again, thank you very much for your comments and suggestions. Yours sincerely, Zhang XiaoWen E-mail: 503212590@qq.com Submitted filename: Response to Reviewers.docx 15 Oct 2025 An algorithm for drug retrieval based on robot-grasping detection constraints and DDPG autonomous learning PONE-D-25-23921R1 Dear Dr. Zhang, We’re pleased to inform you that your manuscript has been judged scientifically suitable for publication and will be formally accepted for publication once it meets all outstanding technical requirements. Within one week, you’ll receive an e-mail detailing the required amendments. When these have been addressed, you’ll receive a formal acceptance letter and your manuscript will be scheduled for publication. An invoice will be generated when your article is formally accepted. Please note, if your institution has a publishing partnership with PLOS and your article meets the relevant criteria, all or part of your publication costs will be covered. Please make sure your user information is up-to-date by logging into Editorial Manager at Editorial Manager® and clicking the ‘Update My Information' link at the top of the page. For questions related to billing, please contact billing support . If your institution or institutions have a press office, please notify them about your upcoming paper to help maximize its impact. If they’ll be preparing press materials, please inform our press team as soon as possible -- no later than 48 hours after receiving the formal acceptance. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. Kind regards, Andrea Tigrini, Ph.D. Academic Editor PLOS ONE Additional Editor Comments (optional): Authros addressed all the concerns and the manuscript can be accepted for publication. Reviewers' comments: Reviewer's Responses to Questions Comments to the Author Reviewer #3: All comments have been addressed Reviewer #4: All comments have been addressed ********** 2. Is the manuscript technically sound, and do the data support the conclusions??> Reviewer #3: Yes Reviewer #4: Yes ********** 3. Has the statistical analysis been performed appropriately and rigorously? -->?> Reviewer #3: N/A Reviewer #4: No ********** 4. Have the authors made all data underlying the findings in their manuscript fully available??> The PLOS Data policy Reviewer #3: Yes Reviewer #4: Yes ********** 5. Is the manuscript presented in an intelligible fashion and written in standard English??> Reviewer #3: Yes Reviewer #4: Yes ********** Reviewer #3: Thank you for your detailed and thoughtful responses to reviewer comments. I appreciate the effort you have made to address the concerns raised. You have effectively incorporated the major suggestions. Overall, I am satisfied with the current version of your manuscript. The revisions have strengthened the work, and in my view, it is now well-prepared for publication. Wishing you success with your ongoing and future research endeavors. Best regards Reviewer #4: Dear Authors, I have found that you've addressed the feedback thoroughly and explicitly including latency clarification, comparitive baselines for algorithms, real world experiments and clarifications of performance gain. The manuscript is technically sound and clearly structured. The problem statement is thoroughly addressed with the proposed methodology. A statistical analysis of the results provided is acceptable, though it could benefit from a deeper quantitative analysis. Overall, I find the manuscript meeting the necessary standards for PLOS One. ********** what does this mean? ). If published, this will include your full peer review and any attached files. If you choose “no”, your identity will remain anonymous but your review may still be made public. Do you want your identity to be public for this peer review? For information about this choice, including consent withdrawal, please see our Privacy Policy Reviewer #3: No Reviewer #4: Yes: Akshay Aggarwal ********** PONE-D-25-23921R1 PLOS ONE Dear Dr. Zhang, I'm pleased to inform you that your manuscript has been deemed suitable for publication in PLOS ONE. Congratulations! Your manuscript is now being handed over to our production team. At this stage, our production department will prepare your paper for publication. This includes ensuring the following: * All references, tables, and figures are properly cited * All relevant supporting information is included in the manuscript submission, * There are no issues that prevent the paper from being properly typeset You will receive further instructions from the production team, including instructions on how to review your proof when it is ready. Please keep in mind that we are working through a large volume of accepted articles, so please give us a few days to review your paper and let you know the next and final steps. Lastly, if your institution or institutions have a press office, please let them know about your upcoming paper now to help maximize its impact. If they'll be preparing press materials, please inform our press team within the next 48 hours. Your manuscript will remain under strict press embargo until 2 pm Eastern Time on the date of publication. For more information, please contact onepress@plos.org. You will receive an invoice from PLOS for your publication fee after your manuscript has reached the completed accept phase. If you receive an email requesting payment before acceptance or for any other service, this may be a phishing scheme. Learn how to identify phishing emails and protect your accounts at https://explore.plos.org/phishing. If we can help with anything else, please email us at customercare@plos.org. Thank you for submitting your work to PLOS ONE and supporting open access. Kind regards, PLOS ONE Editorial Office Staff on behalf of Dr. Andrea Tigrini Academic Editor PLOS ONE