A Performance Comparison of Commercial Speech Recognition APIs in Noisy Environments

This paper compares the performance of five commercial speech recognition APIs under noisy environments, namely those provided by Amazon AWS, Microsoft Azure, Google, Kakao, and Naver. To this end, we used an open dataset for development and evaluation of multi-channel noise processing technology provided in AI Hub. We tested each API's performance with respect to the speaker's gender and location and the speech content, and measured their error rate using both word error rate (WER) and character error rate (CER). Except for the AWS API, the error rate was higher when tested with female's data than male's one, and when tested with the data recorded from the side than the front. The error rate was also relatively high when the test sentences contained proper nouns such as person's names and local names, and the shorter the sentences, the higher the error rate. Moreover, the Google API outperformed all the others in terms of both WER and CER, with 53% and 18% of error rate, respectively. © 2022 Korean Institute of Electrical Engineers. All rights reserved.