ntvНаучно-технический вестник информационных технологий, механики и оптикиScientific and Technical Journal of Information Technologies, Mechanics and Optics2226-14942500-0373Университет ИТМО10.17586/2226-1494-2026-26-4-826-834ntv-650Research ArticleКОМПЬЮТЕРНЫЕ СИСТЕМЫ И ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИCOMPUTER SCIENCEСистема распознавания карельской речи с переключением кода карельский-русскийKarelian speech recognition system with support for Karelian-Russian code-switchinghttps://orcid.org/0000-0002-1264-4458КипятковаИ. С.KipyatkovaI. S.

Кипяткова Ирина Сергеевна — кандидат технических наук, доцент, старший научный сотрудник

sc 36018436400

Санкт-Петербург, 199178

Irina S. Kipyatkova — PhD, Associate Professor, Senior Researcher

sc 36018436400

Saint Petersburg, 199178

dolgushin.m@iias.spb.suhttps://orcid.org/0000-0002-4344-2330ДолгушинМ. Д.DolgushinM. D.

Долгушин Михаил Дмитриевич — младший научный сотрудник

sc 58743617900

Санкт-Петербург, 199178

Mikhail D. Dolgushin — Junior Researcher

sc 58743617900

Saint Petersburg, 199178

dolgushin.m@iias.spb.suhttps://orcid.org/0009-0001-5572-3842КиселеваК. О.KiselevaK. O.

Киселева Ксения Олеговна — младший научный сотрудник

sc 60000649000

Санкт-Петербург, 199178

Kseniia O. Kiseleva — Junior Researcher

sc 60000649000

Saint Petersburg, 199178

kiseleva.k@iias.spb.suhttps://orcid.org/0000-0003-1196-1117КагировИ. А.KagirovI. A.

Кагиров Ильдар Амирович — научный сотрудник

sc 25121369400

Санкт-Петербург, 199178

Ildar A. Kagirov — Scientific Researcher

sc 25121369400

Saint Petersburg, 199178

kagirov@iias.spb.suСанкт-Петербургский Федеральный исследовательский центр Российской академии наукРоссияSt. Petersburg Federal Research Center of the Russian Academy of Sciences (SPC RAS)Russian Federation202627082026264826834Copyright © Кипяткова И.С., Долгушин М.Д., Киселева К.О., Кагиров И.А., 20262026Кипяткова И.С., Долгушин М.Д., Киселева К.О., Кагиров И.А.Kipyatkova I.S., Dolgushin M.D., Kiseleva K.O., Kagirov I.A.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ntv.elpub.ru/jour/article/view/650

Введение. Представлена система автоматического распознавания устной речи для ливвиковского наречия карельского языка в условиях переключения кодов между карельским и русским языками. Выполнено исследование методов автоматического распознавания билингвальной речи. С целью повышения точности распознавания разработана методика аугментации текстовых данных, включающая в себя частичный перевод и синтез словоформ с внутрисловным переключением кода. Метод. Применено акустическое моделирование путем дообучения предобученной многоязычной модели Wav2Vec2-BERT 2.0 на материале двух предварительно собранных баз данных общим объемом 7,5 ч. Дообучение осуществлялось с использованием фреймворка Transformers. При построении языковой модели для решения проблемы нехватки текстовых данных с переключением кода был применен метод аугментации, основанный на частичном автоматическом переводе карельских текстов на русский язык с последующей генерацией словоформ с внутрисловным переключением кода по лингвистическим правилам. На основе разработанных правил сгенерирован список слов с внутрисловным переключением кода для использования при создании модели языка. В ходе экспериментов сравнивались различные конфигурации языковой модели, различающиеся количеством итераций текстовой аугментации (от 1 до 5), содержанием словаря (базовый и включающий все сгенерированные формы), а также интерполяцией с русской языковой моделью. Основные результаты. Проведенные эксперименты показали, что использование полного словаря, включающего сгенерированные гибридные словоформы, дает устойчивое улучшение результатов. Дополнительное снижение количества неправильно распознанных слов до 25,82 % на отладочной и 29 % на тестовой частях базы данных было достигнуто при линейной интерполяции карельской языковой модели с русской (коэффициент интерполяции 0,7). Обсуждение. Выполненные эксперименты подтверждают эффективность разработанной методики создания системы распознавания билингвальной речи. В частности, рекомендуется комбинировать дообучение многоязычных акустических моделей, текстовую аугментацию с морфологическими правилами и интерполяцию языковых моделей. Предложенный подход может быть применен для создания систем распознавания речи для других малоресурсных языков России в условиях несбалансированного двуязычия.

This paper focuses on the development of an automatic speech recognition system for the Livvi-Karelian variety of the Karelian language, as it is spoken under conditions of code-switching between Karelian and Russian. The study of bilingual speech recognition methods is carried out. In order to improve the quality of speech recognition, a methodology for training text data augmentation via partial translation and intra-word code-switched wordforms artificial synthesis was developed. Acoustic modeling was performed by fine-tuning a pre-trained multilingual Wav2Vec2-BERT 2.0 model with the use of the data from two previously collected corpora containing 7.5 hours of speech. Fine-tuning was performed using the Transformers framework. When developing the language model, in order to address the problem of limited code-switching data, an augmentation method was applied based on partial automatic translation of Karelian texts into Russian, followed by the generation of word-forms with intra-word code-switching based on special linguistic rules. On the base of formulated rules, a list of words with intra-word code-switching was generated for a language model. The experiments showed that using a full vocabulary that includes generated hybrid word forms yields a consistent improvement in results. A further reduction in word error rate to 25.82 % on the development set and 29 % on the test portion of the corpus was achieved through linear interpolation of the Karelian language model with the Russian language model (interpolation weight 0.7). The conducted experiments confirm the effectiveness of the developed methodology for developing a bilingual speech recognition system. In particular, it is recommended to combine finetuning of multilingual acoustic models, text augmentation with morphological rules, and language model interpolation. The proposed approach can be applied to developing speech recognition systems for other low-resource languages of Russia spoken in an unbalanced bilingual environment.

автоматическое распознавание речипереключение кодовкарельский языкмалоресурсные языкиаугментация данныхвнутрисловное переключение кодаWav2Vec2-BERT 2.0языковое моделированиеautomatic speech recognitioncode-switchingKarelian languagelow-resource languagesdata augmentationintraword code-switchingWav2Vec2-BERT 2.0language modelingРабота выполнена при финансовой поддержке Российского научного фонда, проект № 24-21-00276, https://rscf.ru/project/24-21-00276/.This research is financially supported by the Russian Science Foundation, project No. 24-21-00276, https://rscf.ru/project/24-21-00276/.ReferencesMabokela K.R., Manamela M.J., Manaileng M. Modeling codeswitching speech on under-resourced languages for language identification // Proc. of the 4th Workshop on Spoken Language Technologies for Under Resourced Languages (SLTU). 2014. P. 225-230.Mabokela K.R., Manamela M.J., Manaileng M. Modeling codeswitching speech on under-resourced languages for language identification. Proc. of the 4th Workshop on Spoken Language Technologies for Under Resourced Languages (SLTU), 2014, pp. 225–230.Bhuvanagirir K., Kopparapu S.K. Mixed language speech recognition without explicit identification of language // American Journal of Signal Processing. 2012. V. 2. N 5. P. 92–97. doi: 10.5923/j.ajsp.20120205.02Bhuvanagirir K., Kopparapu S.K. Mixed language speech recognition without explicit identification of language. American Journal of Signal Processing, 2012, vol. 2, no. 5, pp. 92–97. doi: 10.5923/j.ajsp.20120205.02Agro M.T., Kulkarni A.A., Kadaoui K., Talat Z., Aldarmaki H. Codeswitching in End-to-End automatic speech recognition: A systematic literature review // arXiv. 2025. arXiv:2507.07741. doi: 10.48550/arXiv.2507.07741Agro M.T., Kulkarni A.A., Kadaoui K., Talat Z., Aldarmaki H. Codeswitching in End-to-End automatic speech recognition: A systematic literature review. arXiv, 2025. arXiv:2507.07741. doi: 10.48550/arXiv.2507.07741Hsieh I.-T., Wu C.-H., Wang C.-H. Acoustic and textual data augmentation for code-switching speech recognition in underresourced language // Proc. of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC). 2020. P. 302–307.Hsieh I.-T., Wu C.-H., Wang C.-H. Acoustic and textual data augmentation for code-switching speech recognition in underresourced language. Proc. of the Asia-Pacific Signal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2020, pp. 302–307.Hamed I., Habash N., Abdennadher S., Vu N.T. Investigating lexical replacements for Arabic-English code-switched data augmentation // Proc. of the 6th Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT). 2023. P. 86–100. doi: 10.18653/v1/2023.loresmt-1.7Hamed I., Habash N., Abdennadher S., Vu N.T. Investigating lexical replacements for Arabic-English code-switched data augmentation. Proc. of the 6th Workshop on Technologies for Machine Translation of Low-Resource Languages (LoResMT), 2023, pp. 86–100. doi: 10.18653/v1/2023.loresmt-1.7Hussein A., Chowdhury S.A., Abdelali A., Dehak N., Ali A., Khudanpur S. Textual data augmentation for Arabic-English codeswitching speech recognition // Proc. of the IEEE Spoken Language Technology Workshop (SLT). 2023. P. 777–784. doi: 10.1109/SLT54892.2023.10023313Hussein A., Chowdhury S.A., Abdelali A., Dehak N., Ali A., Khudanpur S. Textual data augmentation for Arabic-English codeswitching speech recognition. Proc. of the IEEE Spoken Language Technology Workshop (SLT), 2023, pp. 777–784. doi: 10.1109/SLT54892.2023.10023313Pratapa A., Bhat G., Choudhury M., Sitaram S., Dandapat S., Bali K. Language modeling for code-mixing: The role of linguistic theory based synthetic data // Proc. of the 56th Annual Meeting of the Association for Computational Linguistics. 2018. V. 1. P. 1543–1553. doi: 10.18653/v1/P18-1143Pratapa A., Bhat G., Choudhury M., Sitaram S., Dandapat S., Bali K. Language modeling for code-mixing: The role of linguistic theory based synthetic data. Proc. of the 56th Annual Meeting of the Association for Computational Linguistics, 2018, vol. 1, pp. 1543-1553. doi: 10.18653/v1/P18-1143Chuang S.-P., Sung T.-W., Lee H.-Y. Training code-switching language model with monolingual data // Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP). 2020. P. 7949–7953. doi: 10.1109/ICASSP40776.2020.9053775Chuang S.-P., Sung T.-W., Lee H.-Y. Training code-switching language model with monolingual data. Proc. of the IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2020, pp. 7949–7953. doi: 10.1109/ICASSP40776.2020.9053775Yilmaz E., van Den Heuvel H., van Leeuwen D.A. Acoustic and textual data augmentation for improved ASR of code-switching speech // Proc. of the 19th Annual Conference of the International Speech Communication Association Interspeech. 2018. P. 1933–1937. doi: 10.21437/Interspeech.2018-52Yilmaz E., van Den Heuvel H., van Leeuwen D.A. Acoustic and textual data augmentation for improved ASR of code-switching speech. Proc. of the 19th Annual Conference of the International Speech Communication Association Interspeech, 2018, pp. 19331937. doi: 10.21437/Interspeech.2018-52Chang C.-T., Chuang S.-P., Lee H.-Y. Code-switching sentence generation by generative adversarial networks and its application to data augmentation // Proc. of the 20th Annual Conference of the International Speech Communication Association Interspeech. 2019. P. 554–558. doi: 10.21437/Interspeech.2019-3214Chang C.-T., Chuang S.-P., Lee H.-Y. Code-switching sentence generation by generative adversarial networks and its application to data augmentation. Proc. of the 20th Annual Conference of the International Speech Communication Association Interspeech, 2019, pp. 554–558. doi: 10.21437/Interspeech.2019-3214Şahin G.G. To augment or not to augment? A comparative study on text augmentation techniques for low-resource NLP // Computational Linguistics. 2022. V. 48. N 1. P. 5–42. doi: 10.1162/coli_a_00425Şahin G.G. To augment or not to augment? A comparative study on text augmentation techniques for low-resource NLP. Computational Linguistics, 2022, vol. 48, no. 1, pp. 5–42. doi: 10.1162/coli_a_00425Wan Z., Wan X., Peng W., Li R. New datasets and controllable iterative data augmentation method for code-switching ASR error correction // Findings of the Association for Computational Linguistics: EMNLP. 2023. P. 8075–8087. doi: 10.18653/v1/2023.findings-emnlp.543Wan Z., Wan X., Peng W., Li R. New datasets and controllable iterative data augmentation method for code-switching ASR error correction. Findings of the Association for Computational Linguistics: EMNLP, 2023, pp. 8075–8087. doi: 10.18653/v1/2023.findingsemnlp.543Kobayashi S. Contextual augmentation: Data augmentation by words with paradigmatic relations // Proc. of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. 2018. V. 2. P. 452–457. doi: 10.18653/v1/N18-2072Kobayashi S. Contextual augmentation: Data augmentation by words with paradigmatic relations. Proc. of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, 2018, vol. 2, pp. 452–457. doi: 10.18653/v1/N18-2072Hamed I., Denisov P., Li C.-Y., Elmahdy M., Abdennadher S, Vu N.T. Investigations on speech recognition systems for low-resource dialectal Arabic–English code-switching speech // Computer Speech & Language. 2022. V. 72. P. 101278. doi: 10.1016/j.csl.2021.101278Hamed I., Denisov P., Li C.-Y., Elmahdy M., Abdennadher S, Vu N.T. Investigations on speech recognition systems for low-resource dialectal Arabic English code-switching speech. Computer Speech & Language, 2022, vol. 72, pp. 101278. doi: 10.1016/j.csl.2021.101278Ubskii D., Matveev Y., Minker W. Impact of using a bilingual model on Kazakh-Russian code-switching speech recognition // CEUR Workshop Proceedings. 2020. V. 2590. P. 1–6.Ubskii D., Matveev Y., Minker W. Impact of using a bilingual model on Kazakh-Russian code-switching speech recognition. CEUR Workshop Proceedings, 2020, vol. 2590, pp. 1–6.Карпов А.А., Кипяткова И.С. Методология оценивания работы систем автоматического распознавания речи // Известия высших учебных заведений. Приборостроение. 2012. T. 55. № 11. С. 38-43.Karpov A.A., Kipyatkova I.S. Methodology for estimation of automatic speech recognition system performance. Journal of Instrument Engineering, 2012, vol. 55, no. 11, pp. 38–43. (in Russian)Arkhangelskiy T. Low-resource ASR with an augmented language model // Proc. of the 7th International Workshop on Computational Linguistics of Uralic Languages. 2021. P. 40–46.Arkhangelskiy T. Low-resource ASR with an augmented language model. Proc. of the 7th International Workshop on Computational Linguistics of Uralic Languages, 2021, pp. 40–46.Wolf T., Debut L., Sanh V., Chaumond J., Delangue C., Moi A., et al. Transformers: State-of-the-art natural language processing // Proc. of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations. 2020. P. 38–45. doi: 10.18653/v1/2020.emnlp-demos.6Wolf T., Debut L., Sanh V., Chaumond J., Delangue C., Moi A., et al. Transformers: State-of-the-art natural language processing. Proc. of the Conference on Empirical Methods in Natural Language Processing: System Demonstrations, 2020, pp. 38–45. doi: 10.18653/v1/2020.emnlp-demos.6Кипяткова И.С., Кагиров И.А., Долгушин М.Д. Применение предварительно обученных многоязычных моделей для распознавания карельской речи // Информатика и автоматизация. 2025. Т. 24. № 2. С. 604–630. doi: 10.15622/ia.24.2.9Kipyatkova I., Kagirov I., Dolgushin M. Use of pre-trained multilingual models for Karelian speech recognition. Informatics and Automation, 2025, vol. 24, no. 2, pp. 604–630. (in Russian). doi: 10.15622/ia.24.2.9Кагиров И.А., Киселева К.О., Кипяткова И.С. Анализ переключения кодов в речи носителей карельского языка // Terra Linguistica. 2025. Т. 16. № 2. С. 22–40. doi: 10.18721/JHSS.16202Kagirov I.A., Kiseleva K.O., Kipyatkova I.S. Code-switching analysis in Karelian language speakers’ speech. Terra Linguistica, 2025, vol. 16, no. 2, pp. 22–40. (in Russian). doi: 10.18721/JHSS.16202Kipyatkova I., Kiseleva K., Dolgushin M., Kagirov I. Modeling intraword code-switching for Karelian ASR // Lecture Notes in Computer Science. 2025. V. 16188. P. 104–117. doi: 10.1007/978-3-032-07959-6_8Kipyatkova I., Kiseleva K., Dolgushin M., Kagirov I. Modeling intraword code-switching for Karelian ASR. Lecture Notes in Computer Science, 2025, vol. 16188, pp. 104–117. doi: 10.1007/978-3-032-07959-6_8Kipyatkova I. Experimenting with hybrid TDNN/HMM acoustic models for Russian speech recognition // Lecture Notes in Computer Science. 2017. V. 10458. P. 362–369. doi: 10.1007/978-3-319-66429-3_35Kipyatkova I. Experimenting with hybrid TDNN/HMM acoustic models for Russian speech recognition. Lecture Notes in Computer Science, 2017, vol. 10458, pp. 362–369. doi: 10.1007/978-3-319-66429-3_35Кипяткова И.С., Родионова А.П., Кагиров И.А., Крижановский А.А. Подготовка речевых и текстовых данных для создания системы автоматического распознавания карельской речи // Ученые записки Петрозаводского государственного университета. 2023. Т. 45. № 5. С. 89–98. doi: 10.15393/uchz.art.2023.924Kipyatkova I.S., Rodionova A.P., Kagirov I.A., Krizhanovsky I.A. Speech and text data preparation for development of an automatic speech recognition system for the Karelian language. Proceedings of Petrozavodsk State University, 2023, vol. 45, no. 5, pp. 89–98. (in Russian). doi: 10.15393/uchz.art.2023.924Kipyatkova I., Karpov A. Lexicon size and language model order optimization for Russian LVCSR // Lecture Notes in Computer Science. 2013. V. 8113. P. 219–226. doi: 10.1007/978-3-319-01931-4_29Kipyatkova I., Karpov A. Lexicon size and language model order optimization for Russian LVCSR. Lecture Notes in Computer Science, 2013, vol. 8113, pp. 219–226. doi: 10.1007/978-3-319-01931-4_29Boyko T., Zaitseva N., Krizhanovskaya N., Krizhanovsky A., Novak I., Pellinen N., et al. The open corpus of the Veps and Karelian languages: Overview and applications // KnE Social Sciences. 2022. P. 29–40. doi: 10.18502/kss.v7i3.10419Boyko T., Zaitseva N., Krizhanovskaya N., Krizhanovsky A., Novak I., Pellinen N., et al. The open corpus of the Veps and Karelian languages: Overview and applications. KnE Social Sciences, 2022, pp. 29–40. doi: 10.18502/kss.v7i3.10419Stolcke A., Zheng J., Wang W., Abrash V. SRILM at sixteen: update and outlook // Proc. of the IEEE Automatic Speech Recognition and Understanding Workshop. 2011. P. 5–9.Stolcke A., Zheng J., Wang W., Abrash V. SRILM at sixteen: update and outlook. Proc. of the IEEE Automatic Speech Recognition and Understanding Workshop, 2011, pp. 5–9.

The authors declare that there are no conflicts of interest present.