Improving radiology report generation with multi-grained abnormality prediction

Conventional data-driven approaches for radiology report generation face a problem that the descriptions of the normal regions in the real data are much more than the that for abnormal ones, which potentially leads to the bias of losing focus on abnormalities. Previous work showed promising results owing to the fact that most of the content in reports are described within normal range, which, although fluent, has the limitation of tending to favor the evaluation metrics rather than produce useful hints for human judgment and model learning. To this end, we propose to explicitly predict abnormalities for radiology report generation, following a multitask learning scheme to drive the model paying more attention on the abnormal regions with multi-grained information, including abnormalities in different granularities, so as to tackle the aforementioned limitation for report generation. In doing so, we propose a disease detector (DD) to identify coarse-grained abnormality, and a medical concept detector (MCD) to associate the predicted disease with the predefined fine-grained pathological concepts. To integrate the information from the proposed abnormality prediction, we design a dual-stream adaptive decoder that takes such information into account with a gate unit controls the integration at each generation step. Extensive experiment results on two widely used benchmark datasets indicate that our method achieves 29.8% and 34.2% performance improvement over the baseline on the natural language generation (NLG) metrics and clinical efficacy (CE) metrics respectively, demonstrate the superiority of the proposed approach.