Learning to decode to future success for multi-modal neural machine translation

Existing only-text NMT (neural machine translation) systems can benefit from explicitly modelling target future contexts as recurrent states. However, the modelled target future context is implicit in the conventional only-text NMT as the target is invisible in inference. For the Multi-modal Neural Machine Translation (MNMT), the visual annotation presents the content described in the bilingual parallel sentence pair, so-called multi-modal consistency. This consistency provides an advantage that future target context can be simulated in visual features. This paper proposes a novel translation model that allows MNMT to encode the future target context from the visual annotation in auto-regressive decoding. Our model uses visual-target consistency to enhance the target generation. Moreover, we use the multi-modal consistency that fully uses the visual annotation to encourage the semantic agreement between bilingual parallel sentences and the pivoted visual annotation. Empirical results on several recent multi-model translation datasets demonstrated the MNMT model which we proposed significantly improved translation performance on a strong baseline, especially achieving new state-of-the-art results on all three language pairs with visual annotations. Our code will be available after acceptance.