The application and challenges of spectral and image two-modal fusion techniques in coal gangue recognition

During coal mining, it is difficult to avoid gangue generation, and the gangue separation techniques based on spectra and images have problems such as one-sided feature description and weak robustness, which can not meet the practical demands and deal with the complex and abnormal working conditions. In recent years, multimodal technology has been developed and successfully applied in many fields, and this progress provides a new solution for the field of coal gangue identification. There is a lack of a systematic analysis and summary on how to use multimodal technology to improve the accuracy of gangue separation by fusing spectral and image information complementarily. This paper proposes the key techniques of spectral and image fusion, including preprocessing and alignment, representation learning, fusion methods, which builds a technical framework for spectral and image fusion. We focus on the fusion methods of spectral and image, which are classified into four major categories in this paper, including conventional fusion, probabilistic and statistical-based fusion, deep learning-based fusion, and decision fusion, and we further provide guidelines for the use of spectral and image fusion, and the test results on the dataset show that the fusion of spectral and image information can significantly improve the recognition of coal gangue. Based on this, we explore the association between spectral and image features in depth, and propose four optimization methods, including feature reuse, channel-level feature fusion, progressive exploration, and attentional mechanism, which indicate the direction of development after fusion. In addition, we discuss the advantages and limitations of each method, clarify the challenges of spectral and image fusion applied to the field of coal gangue separation. This work study is a guidance and reference for the application of image fusion in the field of coal gangue separation.