MFormer: Taming Masked Transformer for Unsupervised Spectral Reconstruction

Spectral reconstruction (SR) aims to recover the hyperspectral images (HSIs) from the corresponding RGB images directly. Most SR studies based on supervised learning require massive data annotations to achieve superior reconstruction performance, which is limited by complicated imaging techniques and laborious annotation calibration in practice. Thus, unsupervised strategies attract the attention of the community, however, existing unsupervised SR works still face a fatal bottleneck from low accuracy. Besides, traditional convolutional neural network (CNN)-based models are good at capturing local features but experience difficulty in global features. To ameliorate these drawbacks, we propose an unsupervised SR architecture with strong constraints, especially constructing a novel Masked Transformer (MFormer) to excavate latent hyperspectral characteristics to restore realistic HSIs further. Concretely, a dual spectralwise multihead self-attention (DSSA) mechanism embedded in transformer is proposed to firmly associate multihead and channel dimensions and then capture the spectral representation in the implicit solution spaces. Furthermore, a plug-and-play mask-guided band augment (MBA) module is presented to extract and further enhance the bandwise correlation and continuity to boost the robustness of the model. Innovatively, a customized loss based on the intrinsic mapping from HSIs to RGB images and the inherent spectral structural similarity is designed to restrain spectral distortion. Extensive experimental results on three benchmarks verify that our MFormer achieves superior performance over other state-of-the-art (SOTA) supervised and unsupervised methods under a no-label training process equally.