PKMT-Net: A pathological knowledge-inspired multi-scale transformer network for subtype prediction of lung cancer using histopathological images

The precise subtyping of lung cancer remains a significant and challenging task in clinical practice, and existing computer-aided diagnostic systems often overlook complex and specialized medical knowledge. In response to these challenges, a Pathological Knowledge-inspired Multi-scale Transformer Network (PKMT-Net) was proposed for predicting lung cancer subtypes using histopathological images. PKMT-Net consists of three key modules: a multi-scale soft segmentation module, a cross-attention module, and a weighted multi-scale fusion module. Initially, the multi-scale soft segmentation module simulated the pathologist's reading of histopathological images at various scales, capturing both macroscopic and microscopic characteristics. This module implements a novel soft patch generation strategy to mitigate semantic information loss. Next, the cross-attention module, equipped with skip connections, emulated the pathologist's way of correlating macroscopic and microscopic tumor characteristics. Lastly, the weighted multi-scale fusion module modeled the pathologist's decision-making process by integrating macroscopic and microscopic characteristics. After iterative training, the PKMT-Net model delivered an outstanding performance, attaining Area Under the Curve (AUC) values of 0.9992 for the training set, 0.9959 for the validation set, and 0.9970 for an unseen test set. Compared to single-scale models, PKMTNet's AUC improved by at least 0.0210. The model's interpretability, clinical utility, as well as the outcomes of ablation studies were evaluated comprehensively. Furthermore, the PKMT-Net model's generalizability was demonstrated through additional datasets. These results underscore the feasibility and high performance of the PKMT-Net for the processing of histopathology images. The supporting codes of this work can be found at: https: //github.com/zzl2022/PKMT-Net.