NIR/RGB image fusion for scene classification using deep neural networks

Near-infrared (NIR) imaging can add very useful data to many visible range image processing applications. In this paper, new fusion techniques are proposed to benefit from the data of both NIR/RGB sensors for the application of scene recognition and classification. Scene recognition and classification is an important and challenging branch of computer vision. Also, image fusion is a very well-known method in image processing. In this paper, fusion of RGB and NIR images is applied to improve the performance of scene classification. The proposed fusion technique is based on modified visual salient points (MVSP). Within this process, each RGB channel is fused with NIR channel based on their visual saliency maps. The fusion parameter space is then searched for the best fusion, based on an error function that is defined to measure the degree of conformity of the fused image with the input channels. This error function is defined on the basis of the distance between pixel intensities of NIR channel and fused image. On the contrary, the gradient difference is applied for the distance between RGB channels and fused image in the error function. After finding the most consent fused image, it is applied to deep convolutional neural networks (DCNNs) to perform scene category classification using the transfer learning method. The experimental results show that the proposed method in the scene classification outperforms the prior works. The results thus illustrate how an optimized NIR/RGB fusion can bring about better classification outcomes.