Optimization of crop 3D point cloud reconstruction strategy based on the multi-view automatic imaging system

Digital photography has provided an economic and convenient way to generate three-dimensional (3D) point clouds for high-throughput crop phenotyping in plant 3D reconstruction. Manual acquisition of multi-view images is time-consuming and labor-intensive, as hundreds of images can be required to generate high-quality 3D point clouds. Multi-view automatic imaging systems can be expected to significantly reduce the cost of labor and time. However, the currently available systems cannot balance the number of cameras, the cost of equipment, the efficiency and accuracy of the 3D reconstruction, as well as the applicability to the complexity of plants. In this study, the automatic imaging system was developed to acquire multi-view images of different crops at various growth stages. 3D point clouds were then generated from the multi-view images acquired by different strategies (i.e., different imaging perspectives) and the number of cameras (1, 2, 3, 4, 6, and 10 camera/cameras) using the SFM (structure from motion)-MVS (multi-view stereo) algorithm. Statistical filtering was used to remove the noise and outliers. The non-plant 3D point clouds were removed using RGB colors. The reconstructed 3D models were aligned to the reference 3D models using the ICP (Iterative Closest Point) algorithm. Hausdorff distance between the two models was calculated to combine the reconstruction time for the evaluation of the precision and efficiency of 3D point clouds reconstruction with different strategies. The 3D point cloud reconstruction strategies were optimized for different crops at different growth stages using the efficiency and precision of reconstruction. The optimization criteria: the average Hausdorff distance was less than or close to 0.20 cm, with the minimum normalization of reconstruction time and Hausdorff distance. The reliability of extracting phenotypic parameters (height, width, convex hull volume, and total surface area of plants) was evaluated from the 3D models reconstructed with different strategies. Plant height and the maximum width were calculated directly using the reconstructed 3D point clouds. The Convex Hull was generated to calculate the Convex Hull Volume. Poisson-disk sampling (Explicit radius=0.5, and MonteCarlo oversampling=20) and Ball Pivoting algorithm were used to reconstruct the 3D mesh model. The total surface area of plants was then calculated using the generated 3D mesh model. The results showed that the time of 3D reconstruction increased, whereas, the Hausdorff distance decreased with the increase of the number of cameras. According to the optimization criteria, three to four cameras were used as the optimal reconstruction strategy for rapeseed at seedling, bolting, flowering, and mature stages, while six cameras were used as the optimal reconstruction strategy for rice at the heading stage, cotton at flowering and boll-setting stages, as well as wheat at jointing and filling stages, and ten cameras were as the optimal reconstruction strategy for wheat at tillering stage, rice at maturity stage, maize roots and rapeseed roots at maturity stage. The reliable phenotypic parameters were obtained using the 3D reconstruction system with no less than four cameras (determination coefficient R2>0.90; relative root mean square error RRMSE≤9%). The optimal reconstruction strategy can greatly contribute to the high-efficiency, low-cost, and high-precision 3D reconstruction and phenotypic parameter extraction of multiple crops at different growth stages. © 2023 Chinese Society of Agricultural Engineering. All rights reserved.