Analysis of multimodal fusion strategies in deep learning for ischemic stroke lesion segmentation on computed tomography perfusion data

Stroke poses a significant risk to human life. Segmenting and immediately treating the stroke core stops its further development, therefore, enhancing the likelihood of survival. Convolutional neural networks (CNN) have been very successful in medical image segmentation, namely in the field of deep learning, and have produced the most advanced outcomes. Multi-modal images provide superior outcomes in the segmentation of stroke lesions compared to single-modal images. The integration of input from several modalities at various levels is crucial in determining performance and producing diverse outcomes in deep learning models that use multimodalities. Further investigation is required to explore the optimal methods for processing multimodal data in CNNs, the influence of fusion on CNN learning, and the effect of fusion strategies on lesions of varying sizes. To examine the impact of a multi-modal fusion method on lesion segmentation, we assessed four models using distinct fusion techniques, including early, late, bottleneck, and hierarchical fusions. This study discusses the various fusion procedures used in segmenting the lesion using computed tomography perfusion data. In addition, both quantitative and qualitative assessments, including deep feature analysis and feature similarity, were conducted to assess the impact of the fusion technique on the model’s performance. Furthermore, we examined the influence of fusion techniques on the size of the lesion. In addition, we analyzed the advantages and disadvantages of several multi-modal fusion systems. Our findings demonstrate that the bottleneck fusion technique got the highest dice score, 0.582, on the Ischemic Stroke Lesion Segmentation 2018 validation data as a result of its capacity to construct complex relationships across several modalities. © The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2024.