Nuclei detection in hepatocellular carcinoma and dysplastic liver nodules in histopathology images using bootstrap regression

Hepatocellular carcinoma (HCC) is the most common primary malignant neoplasm of the liver representing the fifth most common malignancy worldwide. This tumor is more common in men than women, with a ratio of 2.7:1. Unlike HCC, Dysplasia is the precancerous nature of liver nodules and is characterized by cellular and nuclear enlargement, nuclear pleomorphism, and multinucleation. Area based Adaptive Expectation Maximization (EM) uses texture, layout, and context features of cells, and grows clusters to obtain texton maps of nucleus. A discriminative model of nucleus and cytoplastic changes of tumor is built by incorporating texture, layout, and context information efficiently. A bootsrap regression model of nuclei and cytoplastic changes are built by incorporating the aforementioned features efficiently. Mean squared error, Peak Signal to Noise ratio and Dice similarity values are used to evaluate the method's classification performance. The proposed method provides high classification and segmentation accuracy of nucleus and extra nuclear content in HCC and dysplasia, which are exceedingly textured in histopathology images, when compared to Adaptive K means, EM method and the state-of-the-art method, Convolutional Neural Networks (CNN). As texton detection reduces the cluttered background of nuclei, the proposed method would be a convenient mechanism for the classification of nuclei and non-nuclear features. In conclusion, this system can detect more eligible cells of precancerous nature as well as malignant cells even in a cluttered background of nuclei.