Instance segmentation of low contrast and high density cell images using Mask R-CNN

The light microscope image of glioblastoma cell line A172 has the characteristics of low contrast and high density. An improved algorithm based on Mask R-CNN is proposed. The residual neural network of this algorithm is introduced into deformable convolution to enhance the segmentation ability of multi-cell shapes, and at the same time, based on the feature pyramid network, the high-level semantic structure information is transferred to the bottom layer to form dense connections to adapt to the detection of dense feature images. This instance segmentation algorithm has been verified in the human glioblastoma A172 cell line in the LIVECell-2021 data set. Comparative experiments show that our method performs better in COCO evaluation indicators and visual segmentation effects. Among them, in terms of detection performance, AP increased by 0.319%, AP50 increased by 0.107%, and AP50 increased by 0.136% in segmentation performance.