Three-dimensional reconstruction for dragon fruits and branches using semantic segmentation and geometric analysis

Robotics have been rapidly evolving with the onset of complex tasks for the discrete decision-making or subtle handling in modern agriculture, especially the harvesting of fruits from the orchards. Harvesting robots have also limited to an obstacle under an unstructured environment, such as disorderly grown branches. An added layer of complexity can also reduce the successful harvesting rates during operation. In this study, three-dimensional reconstruction of fruit branches was presented to efficiently avoid the obstacles for robots. The path planning was also determined for the target instruction during detection. Firstly, Intel RealSense D435i depth camera was used as image acquisition equipment to collect the images from dragon fruit plantations in Zengcheng District, Conghua District and Panyu District of Guangzhou. The data set was expanded to 3347 images by operations, such as turning, adding noise and blurring. The MobileNetV2 was selected to replace the traditional feature extraction network, Xception in the DeepLabV3+ backbone. A coordinate attention mechanism was incorporated into the feature extraction module to enhance the performance. The mobile-friendly and lightweight architecture of MobileNetV2 was used to most effectively discern and detect data. Each detected piece of information was converted into a three-dimensional point cloud. Then these point clouds were segmented into simply connected regions by region growing. As such, the robot was operated on the actual physical features of the on-field elements onto a digital platform after transformation. Moreover, a hypothesis was then proposed, indicating that dragon fruits were significantly represented by a three-dimensional ellipsoid. An exhaustive study was conducted to verify this presumption using fitting ellipsoids. Non-linear least square estimation was then utilized to reconstruct the dragon fruit point cloud. The irregular branches were enclosed within a finite set of axis-aligned bounding boxes (AABB), in order to obtain their three-dimensional spatial pose information for the robots to avoid around them. Finally, the harvesting system of dragon fruit was constructed with the three-dimensional reconstruction. Several experiments were conducted to validate the reconstruction efficiency and picking competence, including reconstruction error experiments and simulated picking tests. All coordinate axes were performed in the Cartesian space. The remarkable performance was achieved in the improved DeepLabV3+ model, both in terms of mean intersection over union (mIoU) and mean pixel accuracy (mPA) with a score of 95.59% and 98.01%, respectively, indicating an increased accuracy and precision. In terms of technical efficiency, both inference time and the model memory usage were optimized with the average of 94.74 ms and the only 22.52MB, which was 11% of the original model. The improved model increased the accuracy and the general performance of the system. In practical, the minimal average absolute errors were 0.44 and 2.04 mm, respectively, for the dimensions and the depth of the dragon fruit post-reconstruction. Furthermore, the standard deviation of the three-dimensional reconstructions of the branches was also less than 10mm on all axes. The simulation of the picking test was successfully completed in 86.79% of the experiments. The hypothesis was validated to enhance the efficiency and success rate of the dragon fruit harvesting robot. The dragon fruits and the branches were effectively reconstructed for the planning the path of the robotic system in an authentic environment. The successfully harvesting the fruits was realized to avoid the obstacles in an unstructured environment. © 2024 Chinese Society of Agricultural Engineering. All rights reserved.