On the Optimal Interpolation Weights for Hybrid Autoregressive Transducer Model

This paper explores rescoring strategies to improve a two-pass speech recognition system when the first-pass is a hybrid autoregressive transducer model and the second-pass is a neural language model. The main focus is on the scores provided by each of these models, their quantitative analysis, how to improve them and the best way to combine them to achieve better recognition accuracy. Several analyses are presented to emphasize the importance of the choice of the integration weights for combining the first-pass and the second-pass scores. A sequence level combination weight estimation model along with four training criteria are proposed which allows adaptive integration of the scores per acoustic sequence. The effectiveness of this algorithm is demonstrated by constructing and analyzing models on the Librispeech data set. It is shown that the proposed adaptive weight interpolation technique achieves 5 % relative gain over the baseline model with non-adaptive weights.