Effective Multi-Species weed detection in complex wheat fields using Multi-Modal and Multi-View image fusion

Rapid and accurate acquisition of weed information in wheat fields is the precondition and key to precision weeding. Weed detection in open and complex wheat fields faces two challenges: 1) detection of grass weeds that have a similar appearance to wheat; and 2) weed detection under leaf occlusion. Dual-modal information, namely, red-green-blue (RGB) images and depth images can be introduced, which fundamentally overcomes limitations of single-modal image features in identifying grass weeds. Then, a dual-path Swin Transformer model was developed for multi-modal feature extraction. An alignment and attention module (AAM) was designed to realize the alignment and fusion of information in different modalities. Finally, multi-view images were introduced to break the limitation of leaf occlusion in natural wheat fields by completing the feature space of objects, and the multi-view information fusion method based on the common underlying principle of view agreement was proposed. The experimental results demonstrate that depth information is a valuable complement to RGB imagery, facilitating the detection of grass weeds and significantly enhancing weed detection accuracy in wheat fields. Compared with Deep Convolutional Neural Network (DCNN) model, dual-path Swin Transformer model and the model containing AAM have resulted in an improvement in weed detection accuracy by 6.6% and 11.03%, respectively. Additionally, incorporating multi-view information has effectively addressed the issue of leaf occlusion in weed detection, resulting in an 85.14% increase in weed detection accuracy. Furthermore, the problem of view divergence in multi-view learning has been resolved, reducing the false detection rate by 23.94% compared to the direct Combination method.