ntvНаучно-технический вестник информационных технологий, механики и оптикиScientific and Technical Journal of Information Technologies, Mechanics and Optics2226-14942500-0373Университет ИТМО10.17586/2226-1494-2022-22-6-1143-1149ntv-332Research ArticleКОМПЬЮТЕРНЫЕ СИСТЕМЫ И ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИCOMPUTER SCIENCEПовышение точности распознавания внесловарных слов для интегральной системы автоматического распознавания русской речиImproving out of vocabulary words recognition accuracy for an end-to-end Russian speech recognition systemhttps://orcid.org/0000-0002-8697-832XАндрусенкоА. Ю.AndrusenkoA. Yu.

Андрусенко Андрей Юрьевич – аспирант, научный сотрудник; программист

Санкт-Петербург, 194044;

Санкт-Петербург, 197101

sc 57211637170

Andrei Yu. Andrusenko – PhD Student, Scientific Researcher; Software Developer 

Saint Petersburg, 194044;

Saint Petersburg, 197101

sc 57211637170

andrusenkoau@itmo.ruhttps://orcid.org/0000-0001-6267-018XРоманенкоА. Н.RomanenkoA. N.

Романенко Алексей Николаевич – кандидат технических наук, ведущий научный сотрудник; старший научный сотрудник

Санкт-Петербург, 194044;

Санкт-Петербург, 197101

sc 56414341400

Aleksei N. Romanenko – PhD, Leading Researcher; Senior Researcher

Saint Petersburg, 194044;

Saint Petersburg, 197101

sc 56414341400

AlexeySk8@gmai.comООО «ЦРТ-инновации»; Университет ИТМОРоссияSTC-Innovations Ltd.; ITMO UniversityRussian Federation20221812202422611431149Copyright © Андрусенко А.Ю., Романенко А.Н., 20242024Андрусенко А.Ю., Романенко А.Н.Andrusenko A.Y., Romanenko A.N.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ntv.elpub.ru/jour/article/view/332Предмет исследования

Предмет исследования. Системы автоматического распознавания речи (Automatic Speech Recognition, ASR) активно внедряются в нашу повседневную жизнь, тем самым упрощая способ взаимодействия с электронными устройствами. Развитие интегральных (end-to-end) подходов только ускоряет данный процесс. Тем не менее постоянная эволюция и большая степень флективности русского языка приводят к проблеме распознавания новых вне словарных (Out Оf Vocabulary, OOV) слов, которые не принимали участие в процессе обучения ASR-системы при ее создании. В связи с этим ASR-модель может прогнозировать наиболее похожее слово из обучающих данных, что влечет к ошибке распознавания. Особенно это касается ASR-моделей, использующих декодирование на основе взвешенного конечного автомата (Weighted Finite State Transducer, WFST), так как они заведомо ограничены списком словарных слов, которые могут появиться в результате распознавания. Выполнено исследование проблемы на основе открытой базы русского языка (common voice) и интегральной ASR-системы, использующей WFST-декодер.

Метод

Метод. Предложен метод дообучения интегральной ASR-системы на основе дискриминативной функции потерь MMI (Maximum Mutual Information) и метода декодирования интегральной модели с помощью TG графа. Дискриминативное обучение позволило сгладить распределение вероятностей предсказания акустических классов, добавив таким образом большую вариативность в результате распознавания. Так как декодирование с помощью TG графа не имеет ограничений на распознавание только словарных слов, оно позволило использовать языковую модель, обученную на большом количестве внешних текстовых данных.

Основные результаты

Основные результаты. В качестве тестового множества использована восьмичасовая подвыборка из базы common voice. Общее число OOV-слов в тестовой выборке составило 18,1 %. Полученные результаты показали, что использование предложенных методов сократило пословную ошибку распознавания на 3 % в абсолютном значении относительно стандартного метода декодирования интегральных моделей. При этом сохранилась возможность распознавания OOV-слов на сравнимом уровне.

Практическая значимость

Практическая значимость. Использование предложенных методов может улучшить общее качество распознавания ASR-систем и сделать их более устойчивыми к распознаванию новых слов, которые не участвовали в процессе обучения модели.

Automatic Speech Recognition (ASR) systems are experiencing an active introduction into our daily lives, simplifying the way we interact with electronic devices. The advent of end-to-end approaches has only accelerated this process. However, the constant evolution and a high degree of inflection of the Russian language lead to the problem of recognizing new words outside the vocabulary (Out Of Vocabulary, OOV) because they did not take part in the training process of the ASR system. In such a case, the ASR model tends to predict the most similar word from the training data which leads to a recognition error. This is especially true for ASR models that use decoding based on a Weighted Finite State Transducer (WFST), since they are obviously limited by the list of vocabulary words that may appear as a result of recognition. In this paper, this problem is investigated on the basis of an open data set of the Russian language (common voice) and an integrated ASR system using the WFST decoder. A method for retraining an integral ASR system based on the discriminative loss function MMI (maximum mutual information) and a method for decoding the integral model using a TG graph are proposed. Discriminative learning allows smoothing the probability distribution of acoustic class prediction, thus adding more variability in the recognition results. Decoding using the TG graph, in turn, is not limited to recognizing only vocabulary words and allows the use of a language model trained on a large amount of external text data. An eight-hour subset from the common voice base is used as a test set. The total number of OOV words in this test sample is 18.1 %. The results show that the use of the proposed methods allows to reduce the word recognition error (Word Error Rate, WER) by 3 % in absolute value relative to the standard method of decoding integral models (beam search), while maintaining the ability to recognize OOV words at a comparable level. The use of the proposed methods should improve the overall quality of recognition of ASR systems and make such systems more resistant to the recognition of new words that were not involved in the learning process.

автоматическое распознавание речиинтегральные системыдискриминативное обучениеOOV-словавзвешенный конечный автоматautomatic speech recognitionend-to-end ASRdiscriminative trainingOOV wordsweighted finite state transducerReferencesHinton G., Deng L., Yu D., Dahl G.E., Mohamed A., Jaitly N., Senior A., Vanhoucke V., Nguyen P., Sainath T.N., Kingsbury B. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups // IEEE Signal Processing Magazine. 2012. V. 29. N 6. P. 82–97. https://doi.org/10.1109/MSP.2012.2205597Hinton G., Deng L., Yu D., Dahl G.E., Mohamed A., Jaitly N., Senior A., Vanhoucke V., Nguyen P., Sainath T.N., Kingsbury B. Deep neural networks for acoustic modeling in speech recognition: The shared views of four research groups // IEEE Signal Processing Magazine. 2012. V. 29. N 6. P. 82–97. https://doi.org/10.1109/MSP.2012.2205597Graves A., Fernandez S., Gomez F., Schmidhuber J. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks // Proc. of the 23rd International Conference on Machine Learning (ICML). 2006. P. 369–376. https://doi.org/10.1145/1143844.1143891Graves A., Fernandez S., Gomez F., Schmidhuber J. Connectionist temporal classification: Labelling unsegmented sequence data with recurrent neural networks // Proc. of the 23rd International Conference on Machine Learning (ICML). 2006. P. 369–376. https://doi.org/10.1145/1143844.1143891Synnaeve G., Xu Q., Kahn J., Likhomanenko T., Grave E., Pratap V., Sriram A., Liptchinsky V., Collobert R. End-to-end ASR: From supervised to semi-supervised learning with modern architectures // arXiv. 2019. ArXiv:1911.08460. https://doi.org/10.48550/arXiv.1911.08460Synnaeve G., Xu Q., Kahn J., Likhomanenko T., Grave E., Pratap V., Sriram A., Liptchinsky V., Collobert R. End-to-end ASR: From supervised to semi-supervised learning with modern architectures // arXiv. 2019. ArXiv:1911.08460. https://doi.org/10.48550/arXiv.1911.08460Li J., Lavrukhin V., Ginsburg B., Leary R., Kuchaiev O., Cohen J.M., Nguyen H., Gadde R.T. Jasper: An end-to-end convolutional neural acoustic model // Proc. of the 20th Annual Conference of the International Speech Communication Association: Crossroads of Speech and Language (INTERSPEECH). 2019. P. 71–75. https://doi.org/10.21437/Interspeech.2019-1819Li J., Lavrukhin V., Ginsburg B., Leary R., Kuchaiev O., Cohen J.M., Nguyen H., Gadde R.T. Jasper: An end-to-end convolutional neural acoustic model // Proc. of the 20th Annual Conference of the International Speech Communication Association: Crossroads of Speech and Language (INTERSPEECH). 2019. P. 71–75. https://doi.org/10.21437/Interspeech.2019-1819Khokhlov Y., Tomashenko N., Medennikov I., Romanenko A. Fast and accurate OOV decoder on high-level features // Proc. of the 18th Annual Conference of the International Speech Communication Association (INTERSPEECH). 2017. P. 2884–2888. https://doi.org/10.21437/Interspeech.2017-1367Khokhlov Y., Tomashenko N., Medennikov I., Romanenko A. Fast and accurate OOV decoder on high-level features // Proc. of the 18th Annual Conference of the International Speech Communication Association (INTERSPEECH). 2017. P. 2884–2888. https://doi.org/10.21437/Interspeech.2017-1367Alumaë A., Tilk O., Ullah A. Advanced rich transcription system for Estonian speech // Frontiers in Artificial Intelligence and Applications. 2018. V. 307. P. 1–8. https://doi.org/10.3233/978-1-61499-912-6-1Alumaë A., Tilk O., Ullah A. Advanced rich transcription system for Estonian speech // Frontiers in Artificial Intelligence and Applications. 2018. V. 307. P. 1–8. https://doi.org/10.3233/978-1-61499-912-6-1Braun R., Madikeri S., Motlicek P. A comparison of methods for OOV-word recognition on a new public dataset // Proc. of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). 2021. P. 5979–5983. https://doi.org/10.1109/ICASSP39728.2021.9415124Braun R., Madikeri S., Motlicek P. A comparison of methods for OOV-word recognition on a new public dataset // Proc. of the IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). 2021. P. 5979–5983. https://doi.org/10.1109/ICASSP39728.2021.9415124Laptev A., Andrusenko A., Podluzhny I., Mitrofanov A., Medennikov I., Matveev Y. Dynamic acoustic unit augmentation with BPE-dropout for low-resource end-to-end speech recognition // Sensors (Basel). 2021. V. 21. N 9. P. 3063. https://doi.org/10.3390/s21093063Laptev A., Andrusenko A., Podluzhny I., Mitrofanov A., Medennikov I., Matveev Y. Dynamic acoustic unit augmentation with BPE-dropout for low-resource end-to-end speech recognition // Sensors (Basel). 2021. V. 21. N 9. P. 3063. https://doi.org/10.3390/s21093063Andrusenko A., Laptev A., Medennikov I. Exploration of end-to-end ASR for OpenSTT - Russian open speech-to-text dataset // Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2020. V. 12335. P. 35–45. https://doi.org/10.1007/978-3-030-60276-5_4Andrusenko A., Laptev A., Medennikov I. Exploration of end-to-end ASR for OpenSTT - Russian open speech-to-text dataset // Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics). 2020. V. 12335. P. 35–45. https://doi.org/10.1007/978-3-030-60276-5_4An K., Xiang H., Ou Z. CAT: A CTC-CRF based ASR toolkit bridging the hybrid and the end-to-end approaches towards data efficiency and low latency // Proc. of the 21st Annual Conference of the International Speech Communication Association (INTERSPEECH). 2020. P. 566–570. https://doi.org/10.21437/Interspeech.2020-2732An K., Xiang H., Ou Z. CAT: A CTC-CRF based ASR toolkit bridging the hybrid and the end-to-end approaches towards data efficiency and low latency // Proc. of the 21st Annual Conference of the International Speech Communication Association (INTERSPEECH). 2020. P. 566–570. https://doi.org/10.21437/Interspeech.2020-2732Hadian H., Sameti H., Povey D., Khudanpur S. End-to-end speech recognition using lattice-free MMI // Proc. of the 19th Annual Conference of the International Speech Communication, (INTERSPEECH). 2018. P. 12–16. https://doi.org/10.21437/Interspeech.2018-1423Hadian H., Sameti H., Povey D., Khudanpur S. End-to-end speech recognition using lattice-free MMI // Proc. of the 19th Annual Conference of the International Speech Communication, (INTERSPEECH). 2018. P. 12–16. https://doi.org/10.21437/Interspeech.2018-1423Laptev A., Majumdar S., Ginsburg B. CTC variations through new WFST topologies // Proc. of the 23rd Annual Conference of the International Speech Communication Association (INTERSPEECH). 2022. P. 1041–1045 https://doi.org/10.21437/Interspeech.2022-10854Laptev A., Majumdar S., Ginsburg B. CTC variations through new WFST topologies // Proc. of the 23rd Annual Conference of the International Speech Communication Association (INTERSPEECH). 2022. P. 1041–1045 https://doi.org/10.21437/Interspeech.2022-10854Zeyer A., Schlüter R., Ney H. Why does CTC result in peaky behavior? // arXiv. 2021. arXiv:2105.14849. https://doi.org/10.48550/arXiv.2105.14849Zeyer A., Schlüter R., Ney H. Why does CTC result in peaky behavior? // arXiv. 2021. arXiv:2105.14849. https://doi.org/10.48550/arXiv.2105.14849Ardila R., Branson M., Davis K., Henretty M., Kohler M., Meyer J., Morais R., Saunders L., Tyers F.M., Weber G. Common voice: A massively-multilingual speech corpus // Proc. of the 12th International Conference on Language Resources and Evaluation (LREC). 2020. P. 4218–4222.Ardila R., Branson M., Davis K., Henretty M., Kohler M., Meyer J., Morais R., Saunders L., Tyers F.M., Weber G. Common voice: A massively-multilingual speech corpus // Proc. of the 12th International Conference on Language Resources and Evaluation (LREC). 2020. P. 4218–4222.Park D., Chan W., Zhang Y., Chiu C., Zoph B., Cubuk E.D., Le Q.V. SpecAugment: A simple data augmentation method for automatic speech recognition // Proc. of the 20th Annual Conference of the International Speech Communication Association: Crossroads of Speech and Language (INTERSPEECH). 2019. P. 2613–2617 https://doi.org/10.21437/interspeech.2019-2680Park D., Chan W., Zhang Y., Chiu C., Zoph B., Cubuk E.D., Le Q.V. SpecAugment: A simple data augmentation method for automatic speech recognition // Proc. of the 20th Annual Conference of the International Speech Communication Association: Crossroads of Speech and Language (INTERSPEECH). 2019. P. 2613–2617 https://doi.org/10.21437/interspeech.2019-2680Gulati A., Qin J., Chiu C., Parmar N., Zhang Y., Yu J., Han W., Wang S., Zhang Z., Wu Y., Pang R. Conformer: Convolutionaugmented transformer for speech recognition // Proc. of the 21st Annual Conference of the International Speech Communication Association (INTERSPEECH). 2020. P. 5036–5040. https://doi.org/10.21437/Interspeech.2020-3015Gulati A., Qin J., Chiu C., Parmar N., Zhang Y., Yu J., Han W., Wang S., Zhang Z., Wu Y., Pang R. Conformer: Convolutionaugmented transformer for speech recognition // Proc. of the 21st Annual Conference of the International Speech Communication Association (INTERSPEECH). 2020. P. 5036–5040. https://doi.org/10.21437/Interspeech.2020-3015Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I. Attention is all you need // Proc. of the 31st Annual Conference on Neural Information Processing Systems (NIPS). 2017. P. 5998–6008.Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A.N., Kaiser Ł., Polosukhin I. Attention is all you need // Proc. of the 31st Annual Conference on Neural Information Processing Systems (NIPS). 2017. P. 5998–6008.Povey D., Ghoshal A., Boulianne G., Burget L., Glembek O., Goel N., Hannemann M., Motlicek P., Qian Y., Schwarz P., Silovsky J., Stemmer G., Vesely K. The Kaldi speech recognition toolkit // Proc. of the IEEE 2011 Workshop on Automatic Speech Recognition and Understanding. 2011.Povey D., Ghoshal A., Boulianne G., Burget L., Glembek O., Goel N., Hannemann M., Motlicek P., Qian Y., Schwarz P., Silovsky J., Stemmer G., Vesely K. The Kaldi speech recognition toolkit // Proc. of the IEEE 2011 Workshop on Automatic Speech Recognition and Understanding. 2011.Watanabe S., Hori T., Karita S., Hayashi T., Nishitoba J., Unno Y., Soplin N.E.Y., Heymann J., Wiesner M., Chen N., Renduchintala A., Ochiaiet T. ESPnet: End-to-end speech processing toolkit // Proc. of the 19th Annual Conference of the International Speech Communication (INTERSPEECH). 2018. P. 2207–2211. https://doi.org/10.21437/Interspeech.2018-1456Watanabe S., Hori T., Karita S., Hayashi T., Nishitoba J., Unno Y., Soplin N.E.Y., Heymann J., Wiesner M., Chen N., Renduchintala A., Ochiaiet T. ESPnet: End-to-end speech processing toolkit // Proc. of the 19th Annual Conference of the International Speech Communication (INTERSPEECH). 2018. P. 2207–2211. https://doi.org/10.21437/Interspeech.2018-1456

The authors declare that there are no conflicts of interest present.