Combining Topic Information and Structure Information in a Dynamic Language Model

We present a language model implemented with dynamic Bayesian networks that combines topic information and structure information to capture long distance dependencies between the words in a text while maintaining the robustness of standard n-gram models. We show that the model is an extension of sentence level mixture models, thereby providing a Bayesian explanation for these models. We describe a procedure for unsupervised training of the model. Experiments show that it reduces perplexity by 13% compared to an interpolated trigram.