REEDS: Relevance and enhanced entropy based Dempster Shafer approach for next word prediction using language model

Next word prediction (NWP) is an acute problem in the arena of natural language processing. The recent approaches are solely based on the probability distribution of the Language Model. Word prediction is the problem of guessing which word is likely to continue a given initial text fragment. A Language Model consists of a number of documents and each document consists of a set of words. Each document will generate a group of evidence based on the information contained in it. The main task is to fuse the evidence to get a reasonable result. In this paper, a novel relevance and enhanced Deng entropy based Dempster's combination rule is proposed where various documents act as evidence source namely relevance and enhanced entropy based Dempster Shafer approach (REEDS) to predict the next probable word from the Language Model. The rationale behind this approach is to handle the conflicting evidence, reduce computational burden and eradicate long learning time of the neural network. The proposed technique can efficiently manage high conflicting evidence and upgrade the performance of convergence. A numerical example is given to demonstrate the efficiency of the proposed method with other combination schemes. Moreover, the proposed method is implemented in Matlab and the outcomes are compared with feedforward neural network on the same Language Model for validation of predicted next word by the proposed method. The proposed method, REEDS is better than neural approach as it relieves from the computational burden and prolonged learning time. (C) 2019 Elsevier B.V. All rights reserved.