ntvНаучно-технический вестник информационных технологий, механики и оптикиScientific and Technical Journal of Information Technologies, Mechanics and Optics2226-14942500-0373Университет ИТМО10.17586/2226-1494-2024-24-5-779-787ntv-146Research ArticleКОМПЬЮТЕРНЫЕ СИСТЕМЫ И ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИCOMPUTER SCIENCEViSL model: модель автоматической генерации предложений вьетнамского языка жестовViSL model: The model automatically generates sentences of Vietnamese sign languagehttps://orcid.org/0009-0009-5882-7653ХаньДангKhanhDang

Данг Хань - аспирант

Санкт-Петербург, 197101

Khanh Dang - PhD Student

Saint Petersburg, 197101

dangkhanhmta.2020@gmail.comhttps://orcid.org/0000-0001-6711-6399БессмертныйИ. А.BessmertnyI. A.

Бессмертный Игорь Александрович - доктор технических наук, профессор, профессор

Санкт-Петербург, 197101

Igor A. Bessmertny - D.Sc., Full Professor

Saint Petersburg, 197101

bessmertny@itmo.ruУниверситет ИТМОРоссияITMO UniversityRussian Federation202413122024245779787Copyright © Хань Д., Бессмертный И.А., 20242024Хань Д., Бессмертный И.А.Khanh D., Bessmertny I.A.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ntv.elpub.ru/jour/article/view/146

Введение. Основной проблемой при построении интеллектуальных систем является недостаточность данных для машинного обучения, что особенно актуально для распознавания языка жестов для глухих и слабослышащих людей. Одним из способов увеличения объема данных для обучения интеллектуальных систем является их синтез. В отличие от синтеза речи, создавать последовательность жестов на вьетнамском и некоторых других языках, в точности повторяющих текст, невозможно. Это связано с существенной ограниченностью словаря жестов и отличающимся порядком слов в предложениях. Целью работы является обогащение обучающего набора видеоданных для создания систем распознавания вьетнамского языка жестов (Vietnamese Sign Language, ViSL).Метод. Поскольку транслировать слова исходного текста в жесты невозможно, возникает задача перевода с обычного языка на жестовый. Для решения поставленной задачи в работе использован двухфазный процесс. На первой фазе выполняется предварительная обработка текста со стандартизацией текстового формата, сегментацией слов и предложений, а затем кодирование слов с помощью словаря языка жестов. На данном этапе не требуется удалять знаки препинания и стоп-слова, поскольку они связаны с точностью N-граммовой модели. На второй фазе вместо использования синтаксического анализа применяется статистический метод формирования последовательности жестов. При этом за основу берется марковская модель на графе переходов между словами, в которой вероятность следующего слова зависит только от двух предыдущих слов. Вероятности переходов вычисляются на существующем размеченном наборе ViSL. Метод графового поиска в ширину используется для составления списка всех предложений, сгенерированных на основе заданного грамматического правила и матрицы семантического взаимодействия между словами. Обратное значение логарифма произведения вероятности совместного появления последовательных словосочетаний из трех слов в предложении используется для оценки частоты встречаемости этого предложения в заданном наборе данных.Основные результаты. Основываясь на данных ViSL, состоящих из 3234 слов, рассчитаны матрицы вероятности, представляющие отношения между словами, на основе данных ViSL с 50 млн предложений, собранных из вьетнамских газет и журналов. Для различных грамматических правил выполнено сравнение количества сгенерированных предложений и оценка точности 50 наиболее часто встречающихся предложений. Средняя точность составила 88 %. Точность сгенерированных предложений оценена статистическими методами. Показано, что число сгенерированных предложений зависит от количества частей слова, которые помечены в соответствии с правилами грамматики. Семантическая точность сгенерированных предложений высока, если поисковые слова помечены правильными частями речи.Обсуждение. По сравнению с методами машинного обучения, предлагаемая модель дает хорошие результаты для языков без словоизменений и порядка слов, следующих определенным правилам, таких как вьетнамский язык, и не требует больших вычислительных ресурсов. Недостатком модели является зависимость точности от типа слова, предложения и сегментации слов. Взаимосвязь слов зависит от наблюдаемого набора данных. Будущее направление исследований — создание абзацев на языке жестов. Полученные данные могут быть использованы в моделях машинного обучения для задач обработки языка жестов.

The main problem in building intelligent systems is the lack of data for machine learning, which is especially important for sign language recognition for the deaf and hard of hearing. One of the ways to increase the amount of data for training is synthesis. Unlike speech synthesis, it is impossible to create a sequence of gestures in Vietnamese and some other languages that exactly repeat the text. This is due to the significant limitations of the gesture dictionary and the different word order in sentences. The aim of the work is to enrich the educational corpus of video data for use in creating recognition systems for the Vietnamese Sign Language (ViSL). Since it is impossible to translate the words of the source text into gestures one to one, the problem of translating from a regular language into a sign language arises. The paper proposes to use a two-phase process for this. The first phase involves pre-processing the text with standardization of the text format, segmentation of words and sentences, and then encoding the words using the sign language dictionary. At this stage, it should be noted that there is no need to remove punctuation marks and stop words, since they are related to the accuracy of the N-gram model. Next, instead of using syntactic analysis, a statistical method for forming a sequence of gestures is used, and the Markov model on the transition graph between words is taken as a basis in which the probability of the next word depends only on the two previous words. Transition probabilities are calculated on the existing marked corpus of the ViSL. The Breadth-first Search method is used to compile a list of all sentences generated based on a given grammatical rule and a matrix of semantic interactions between words. The inverse of the logarithm of the product of the probabilities of co-occurrence of consecutive 3-word phrases in a sentence is used to estimate the frequency of occurrence of that sentence in a given data set. Based on the ViSL data of 3,234 words, we calculated probability matrices representing the relationships between words based on Vietnamese natural language data with 50 million sentences collected from Vietnamese newspapers and magazines. For different grammar rules, we compare the number of generated sentences and evaluate the accuracy of the 50 most frequent sentences. The average accuracy is 88 %. The accuracy of the generated sentences is estimated by manual statistical methods. The number of generated sentences depends on the number of word parts that are labeled according to the grammar rules. The semantic accuracy of the generated sentences will be very high if the search words are labeled with the correct part-of-speech tagging. Compared with machine learning methods, our proposed method gives very good results for languages without inflections and word order that follow certain rules, such as Vietnamese, and does not require large computational resources. The disadvantage of this method is that its accuracy largely depends on the type of word, sentence, and word segmentation. The relationship of words depends on the observed dataset. Future research direction is to generate paragraphs in sign language. The obtained data can be used in machine learning models for sign language processing tasks.

вьетнамский язык жестовмодель языка жестовавтоматически генерация предложенийn-грамммодель Марковаметод графового поиска в ширинуобогащение данныхграмматические правилаVietnamese sign languagesign language modelautomatic sentence generationn-gramMarkov modelbreadth-first searchdata enrichmentgrammatical rulesReferencesKatti R.K., Sujatha C., Desai P., Shankar G. Character and word level gesture recognition of indian sign language. Proc. of the 2023 IEEE 8th International Conference for Convergence in Technology (I2CT), 2023, pp. 1–6. https://doi.org/10.1109/I2CT57861.2023.10126314Katti R.K., Sujatha C., Desai P., Shankar G. Character and word level gesture recognition of indian sign language // Proc. of the 2023 IEEE 8th International Conference for Convergence in Technology (I2CT). 2023. P. 1–6. https://doi.org/10.1109/I2CT57861.2023.10126314Naz N., Sajid H., Ali S., Hasan O., Ehsan M.K. Signgraph: An efficient and accurate pose-based graph convolution approach toward sign language recognition. IEEE Access, 2023, vol. 11, pp. 19135–19147. https://doi.org/10.1109/ACCESS.2023.3247761Naz N., Sajid H., Ali S., Hasan O., Ehsan M.K. Signgraph: An efficient and accurate pose-based graph convolution approach toward sign language recognition // IEEE Access. 2023. V. 11. P. 19135–19147. https://doi.org/10.1109/ACCESS.2023.3247761Boháček M., Hrúz M. Sign pose-based transformer for word-level sign language recognition. Proc. of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW), 2022, pp. 182–191. https://doi.org/10.1109/WACVW54805.2022.00024Boháček M., Hrúz M. Sign pose-based transformer for word-level sign language recognition // Proc. of the 2022 IEEE/CVF Winter Conference on Applications of Computer Vision Workshops (WACVW). 2022. P. 182–191. https://doi.org/10.1109/WACVW54805.2022.00024Jiang Y., Li F., Li Z., Liu Z., Wang Z. Enhancing continuous sign language recognition with Self-Attention and MediaPipe Holistic. Proc. of the 2023 8th International Conference on Instrumentation, Control, and Automation (ICA), 2023, pp. 97–102. https://doi.org/10.1109/ICA58538.2023.10273118Jiang Y., Li F., Li Z., Liu Z., Wang Z. Enhancing continuous sign language recognition with Self-Attention and MediaPipe Holistic // Proc. of the 2023 8th International Conference on Instrumentation, Control, and Automation (ICA). 2023. P. 97–102. https://doi.org/10.1109/ICA58538.2023.10273118Nayan N., Ghosh D., Pradhan P.M. An unsupervised learning approach to handle movement epenthesis in continuous sign language recognition. Proc. of the 2022 17th International Conference on Control, Automation, Robotics and Vision (ICARCV), 2022, pp. 862–867. https://doi.org/10.1109/ICARCV57592.2022Nayan N., Ghosh D., Pradhan P.M. An unsupervised learning approach to handle movement epenthesis in continuous sign language recognition // Proc. of the 2022 17th International Conference on Control, Automation, Robotics and Vision (ICARCV). 2022. P. 862–867. https://doi.org/10.1109/ICARCV57592.2022.10004317Tran K.B., Nguyen U.D., Huynh Q.T. Continuous sign language recognition using MediaPipe. Proc. of the 2023 International Conference on Advanced Technologies for Communications (ATC), 2023, pp. 493–498. https://doi.org/10.1109/ATC58710.2023.10318855Tran K.B., Nguyen U.D., Huynh Q.T. Continuous sign language recognition using MediaPipe // Proc. of the 2023 International Conference on Advanced Technologies for Communications (ATC). 2023. P. 493–498. https://doi.org/10.1109/ATC58710.2023.10318855Quach L.-D., Nguyen C.-N. Conversion of the Vietnammese grammar into sign language structure using the example-based machine translation algorithm. Proc. of the 2018 International Conference on Advanced Technologies for Communications (ATC), 2018, pp. 27–31. https://doi.org/10.1109/ATC.2018.8587584Quach L.-D., Nguyen C.-N. Conversion of the Vietnammese grammar into sign language structure using the example-based machine translation algorithm // Proc. of the 2018 International Conference on Advanced Technologies for Communications (ATC). 2018. P. 27–31. https://doi.org/10.1109/ATC.2018.8587584Kagirov I., Ryumin D., Ivanko D., Axyonov A., Karpov A. Russian sign language: History, grammar and sociolinguistic situation in brief. Proc. of the Language Technologies for All (LT4All), 2019, pp. 71–74.Kagirov I., Ryumin D., Ivanko D., Axyonov A., Karpov A. Russian sign language: History, grammar and sociolinguistic situation in brief // Proc. of the Language Technologies for All (LT4All). 2019. P. 71–74.Singh C., Bansal R.K., Bansal S. Machine translation techniques using AI: A review. Proc. of the 2023 IEEE International Conference on Computer Vision and Machine Intelligence (CVMI), 2023, pp. 1–5. https://doi.org/10.1109/CVMI59935.2023.10464455Singh C., Bansal R.K., Bansal S. Machine translation techniques using AI: A review // Proc. of the 2023 IEEE International Conference on Computer Vision and Machine Intelligence (CVMI). 2023. P. 1–5. https://doi.org/10.1109/CVMI59935.2023.10464455Tan M., Chen D., Li Z., Wang P. Spelling error correction with BERT based on character-phonetic. Proc. of the 2020 IEEE 6th International Conference on Computer and Communications (ICCC), 2020, pp. 1146–1150. https://doi.org/10.1109/ICCC51575.2020.9345276Tan M., Chen D., Li Z., Wang P. Spelling error correction with BERT based on character-phonetic // Proc. of the 2020 IEEE 6th International Conference on Computer and Communications (ICCC). 2020. P. 1146–1150. https://doi.org/10.1109/ICCC51575.2020.9345276Huang C., Feng Y., Zhang Y., Zhang W. Knowledge Base System of Electrical equipment management and potential risk control based on natural language processing technology. Proc. of the 2023 Asia-Europe Conference on Electronics, Data Processing and Informatics (ACEDPI), 2023, pp. 439–445. https://doi.org/10.1109/ACEDPI58926.2023.00090Huang C., Feng Y., Zhang Y., Zhang W. Knowledge Base System of Electrical equipment management and potential risk control based on natural language processing technology // Proc. of the 2023 Asia-Europe Conference on Electronics, Data Processing and Informatics (ACEDPI). 2023. P. 439–445. https://doi.org/10.1109/ACEDPI58926.2023.00090Liu S., Tang R., Chai J. A news automatic tagging method based on statistical language model. Proc. of the 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI), 2017, pp. 1–5. https://doi.org/10.1109/CISP-BMEI.2017.8302092Liu S., Tang R., Chai J. A news automatic tagging method based on statistical language model // Proc. of the 2017 10th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics (CISP-BMEI). 2017. P. 1–5. https://doi.org/10.1109/CISP-BMEI.2017.8302092Xiao J., Zhou Z. Research Progress of RNN Language Model. Proc. of the 2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA), 2020, pp. 1285–1288. https://doi.org/10.1109/ICAICA50127.2020.9182390Xiao J., Zhou Z. Research Progress of RNN Language Model // Proc. of the 2020 IEEE International Conference on Artificial Intelligence and Computer Applications (ICAICA). 2020. P. 1285–1288. https:// doi.org/10.1109/ICAICA50127.2020.9182390Ganai F., Khursheed F. Predicting next Word using RNN and LSTM cells: Stastical Language Modeling. Proc. of the 2019 Fifth International Conference on Image Information Processing (ICIIP), 2019, pp. 469–474. https://doi.org/10.1109/ICIIP47207.2019.8985885Ganai F., Khursheed F. Predicting next Word using RNN and LSTM cells: Stastical Language Modeling // Proc. of the 2019 Fifth International Conference on Image Information Processing (ICIIP). 2019. P. 469–474. https://doi.org/10.1109/ICIIP47207.2019.8985885Acheampong F.A., Nunoo-Mensah H., Chen W. Recognizing emotions from texts using an ensemble of transformer-based language models. Proc. of the 2021 18th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP), 2021, pp. 161–164. https://doi.org/10.1109/ICCWAMTIP53232.2021.9674102Acheampong F.A., Nunoo-Mensah H., Chen W. Recognizing emotions from texts using an ensemble of transformer-based language models // Proc. of the 2021 18th International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP). 2021. P. 161–164. https://doi.org/10.1109/ICCWAMTIP53232.2021.9674102Lee H., Kim J.-H., Hwang E.J., Kim J., Park J.C. Leveraging large language models with vocabulary sharing for sign language translation. Proc. of the 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW), 2023, pp. 1–5. https://doi.org/10.1109/ICASSPW59220.2023.10193533Lee H., Kim J.-H., Hwang E.J., Kim J., Park J.C. Leveraging large language models with vocabulary sharing for sign language translation // Proc. of the 2023 IEEE International Conference on Acoustics, Speech, and Signal Processing Workshops (ICASSPW). 2023. P. 1–5. https://doi.org/10.1109/ICASSPW59220.2023.10193533Garg H., Gupta I., Kumar K., Kaur B., Pundir D. Artificial intelligence based dynamic approach to visualize the graphs. Proc. of the 2023 International Conference on Computational Intelligence, Communication Technology and Networking (CICTN), 2023, pp. 663–667. https://doi.org/10.1109/CICTN57981.2023.10140873Garg H., Gupta I., Kumar K., Kaur B., Pundir D. Artificial intelligence based dynamic approach to visualize the graphs // Proc. of the 2023 International Conference on Computational Intelligence, Communication Technology and Networking (CICTN). 2023. P. 663–667. https://doi.org/10.1109/CICTN57981.2023.10140873

The authors declare that there are no conflicts of interest present.