Compressing Speech Recognition Networks with MLP via Tensor-Train Decomposition

Deep neural networks (DNNs) have produced state-of-the-art performance in automatic speech recognition (ASR). This success is often associated with a large DNN structure with millions or even billions of parameters. Such large-scale networks take large disk space and require huge computational resources at run-time, therefore not suitable for applications in mobile or wearable devices. In this paper, we investigate a compression approach for DNNs based on Tensor-Train (TT) decomposition and apply it to the ASR task. Our results on the TIMIT database reveals that the compressed networks can maintain the performance of the original full-connected network, while greatly reducing the number of parameters. In particular, we found that the rate of model size decreasing is much larger than the rate of WER (word error rate) increasing, which means that the performance loss caused by the TT-based compression can be well compensated by the model size reduction. Moreover, how many layers and which layer can be substituted by TT is application dependent and should be carefully designed according to the application scenario.