A multimodal medical image contrastive learning algorithm with domain adaptive denormalization

最近，深度学习在医学图像任务中取得了令人瞩目的成果。然而，这种方法通常需要大规模的标注数据，而医学图像的标注成本较高，因此如何从有限的标注数据中进行高效学习是一个难题。目前，常用的两种方法是迁移学习和自监督学习，然而这两种方法在多模态医学图像中的研究却很少，因此本研究提出了一种多模态医学图像对比学习方法。该方法将同一患者不同模态的图像作为正样本，有效增加训练过程中的正样本数量，有助于模型充分学习病灶在不同模态图像上的相似性和差异性，从而提高模型对医学图像的理解能力和诊断准确率。常用的数据增强方法并不适合多模态图像，因此本文提出了一种域自适应反标准化方法，借助目标域的统计信息对源域图像进行转换。本研究以两个不同的多模态医学图像分类任务对本文方法展开验证：在微血管浸润识别任务中，本文方法获得了（74.79 ± 0.74）%的准确率和（78.37 ± 1.94）%的F1分数，相比其它较为熟知的学习方法有所提升；对于脑肿瘤病理分级任务，本文方法也取得了明显的改进。结果表明，本文方法在多模态医学图像数据上取得了良好的结果，可为多模态医学图像的预训练提供一种参考方案。.; Recently, deep learning has achieved impressive results in medical image tasks. However, this method usually requires large-scale annotated data, and medical images are expensive to annotate, so it is a challenge to learn efficiently from the limited annotated data. Currently, the two commonly used methods are transfer learning and self-supervised learning. However, these two methods have been little studied in multimodal medical images, so this study proposes a contrastive learning method for multimodal medical images. The method takes images of different modalities of the same patient as positive samples, which effectively increases the number of positive samples in the training process and helps the model to fully learn the similarities and differences of lesions on images of different modalities, thus improving the model's understanding of medical images and diagnostic accuracy. The commonly used data augmentation methods are not suitable for multimodal images, so this paper proposes a domain adaptive denormalization method to transform the source domain images with the help of statistical information of the target domain. In this study, the method is validated with two different multimodal medical image classification tasks: in the microvascular infiltration recognition task, the method achieves an accuracy of (74.79 ± 0.74)% and an F1 score of (78.37 ± 1.94)%, which are improved as compared with other conventional learning methods; for the brain tumor pathology grading task, the method also achieves significant improvements. The results show that the method achieves good results on multimodal medical images and can provide a reference solution for pre-training multimodal medical images.