ntvНаучно-технический вестник информационных технологий, механики и оптикиScientific and Technical Journal of Information Technologies, Mechanics and Optics2226-14942500-0373Университет ИТМО10.17586/2226-1494-2022-22-3-559-566ntv-272Research ArticleКОМПЬЮТЕРНЫЕ СИСТЕМЫ И ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИCOMPUTER SCIENCEПовышение эффективности обработки жестового языка посредством малокадрового машинного обученияImproving sign language processing via few-shot machine learninghttps://orcid.org/0000-0001-7777-6972ШовкоплясГ. Ф.ShovkopliasG. F.

Шовкопляс Григорий Филиппович — инженер

Санкт-Петербург, 197101

sc 57222048908

Grigory F. Shovkoplias — Engineer

Saint Petersburg, 197101

gfshovkoplias@itmo.ruhttps://orcid.org/0000-0003-1924-0621СтроковД. А.StrokovD. A.

Строков Дмитрий Андреевич — студент

Санкт-Петербург, 197101

Dmitriy A. Strokov — Student

Saint Petersburg, 197101

dmitry.strokov@mail.ruhttps://orcid.org/0000-0001-7974-0922КазанцевД. В.KasantsevD. V.

Казанцев Даниил Владимирович — старший лаборант

Санкт-Петербург, 197101

Daniil V. Kasantsev —Senior Laboratory Assistant

Saint Petersburg, 197101

kazantsev@itkvar.ruhttps://orcid.org/0000-0002-5483-716XВатьянА. С.VatianA. S.

Ватьян Александра Сергеевна — доцент

Санкт-Петербург, 197101

Aleksandra S. Vatian — Associate Professor

Saint Petersburg, 197101

alexvatyan@gmail.comhttps://orcid.org/0000-0002-2581-935XАсадулаевА. А.AsadulaevA. A.

Асадулаев Арип Амирханович — ассистент

Санкт-Петербург, 197101

Arip A. Asadulaev — Assistant

Saint Petersburg, 197101

aripasadulaev@itmo.ruhttps://orcid.org/0000-0003-1886-2867ТомиловИ. В.TomilovI. V.

Томилов Иван Вячеславович — старший лаборант

Санкт-Петербург, 197101

Ivan V. Tomilov — Senior Laboratory Assistant

Saint Petersburg, 197101

ivan-tomilov3@yandex.ruhttps://orcid.org/0000-0002-2723-2077ШалытоА. А.ShalytoA. A.

Шалыто Анатолий Абрамович — доктор технических наук, профессор, профессор

Санкт-Петербург, 197101

sc 57222048908

Anatoly A. Shalyto — D. Sc., Full Professor

Saint Petersburg, 197101

anatoly.shalyto@gmail.comhttps://orcid.org/0000-0002-1361-6037ГусароваН. В.GusarovaN. F.

Гусарова Наталия Федоровна — кандидат технических наук, старший научный сотрудник, доцент

Санкт-Петербург, 197101

Natalia F. Gusarova — PhD, Senior Researcher, Associate Professor

Saint Petersburg, 197101

natfed@list.ruУниверситет ИТМОРоссияITMO UniversityRussian Federation202217122024223559566Copyright © Шовкопляс Г.Ф., Строков Д.А., Казанцев Д.В., Ватьян А.С., Асадулаев А.А., Томилов И.В., Шалыто А.А., Гусарова Н.В., 20242024Шовкопляс Г.Ф., Строков Д.А., Казанцев Д.В., Ватьян А.С., Асадулаев А.А., Томилов И.В., Шалыто А.А., Гусарова Н.В.Shovkoplias G.F., Strokov D.A., Kasantsev D.V., Vatian A.S., Asadulaev A.A., Tomilov I.V., Shalyto A.A., Gusarova N.F.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ntv.elpub.ru/jour/article/view/272Предмет исследования

Предмет исследования. Повышение эффективности коммуникации глухих и слабослышащих людей путем обработки жестового языка средствами искусственного интеллекта – важная задача в социальном и в технологическом планах. Одним из направлений решения этой проблемы является применение достаточно дешевого и доступного маркерного метода.

Метод

Метод. Метод основан на регистрации электромиографических сигналов мышц, регистрируемых с помощью браслета, надеваемого на руку. Для повышения качества распознавания жестов применяется модификация маркерного метода, сущность которого состоит в дублировании датчиков электромиографических сигналов в сочетании с малокадровым подходом машинного обучения.

Основные результаты

Основные результаты. Экспериментально изучены возможности повышения качества обработки жестового языка за счет дублирования электромиографических датчиков и уменьшения набора данных, необходимого для машинного обучения. Выполнено сравнение нескольких технологий малокадрового подхода. Показано, что при обучении малокадровых нейронных сетей на наборе данных объемом 56 000 образцов можно достичь лучших результатов, чем при обучении классификатора типа «случайный лес» на наборе данных объемом 160 000 образцов.

Практическая значимость

Практическая значимость. Использование минимального количества датчиков электромиографических сигналов в сочетании с малокадровыми методами их обработки обеспечивает возможность организации быстрого и экономичного взаимодействия с людьми с нарушениями слуха и речи.

Improving the efficiency of communication of deaf and hard of hearing people by processing sign language using artificial intelligence is an important task both socially and technologically. One of the ways to solve this problem is a fairly cheap and accessible marker method. The method is based on the registration of electromyographic (EMG) muscle signals using bracelets worn on the arm. To improve the quality of recognition of gestures recorded by the marker method, a modification of the marker method is proposed — duplication of EMG sensors in combination with a lowframe machine learning approach. We experimentally study the possibilities of improving the quality of processing of sign language by duplicating EMG sensors as well as by reducing the volume of the dataset required for training machine learning tools. In the latter case, we compare several technologies of the few-shot approach. Our experiments show that training with few-shot neural nets on 56k samples we can achieve better results than training on random forest with 160k samples. The use of a minimum number of sensors in combination with few-shot signal processing techniques provides the possibility of organizing quick and cost-effective interaction with people with hearing and speech disabilities.

обработка жестового языкамалокадровое машинное обучениебраслетымаркерные методыsign language processingfew-shot machine learningbraceletsmarker methodsРабота поддержана Федеральной программой «Приоритет 2030».This research was supported by Priority 2030 Federal Academic Leadership Program.ReferencesAgrawal S.C., Jalal A.S., Tripathi R.K. A survey on manual and nonmanual sign language recognition for isolated and continuous sign // International Journal of Applied Pattern Recognition. 2016. V. 3. N 2. P. 99–134. https://doi.org/10.1504/ijapr.2016.079048Agrawal S.C., Jalal A.S., Tripathi R.K. A survey on manual and nonmanual sign language recognition for isolated and continuous sign // International Journal of Applied Pattern Recognition. 2016. V. 3. N 2. P. 99–134. https://doi.org/10.1504/ijapr.2016.079048Bragg D., Koller O., Bellard M., Berke L., Boudrealt P., Braffort A., Caselli N., Huenerfauth M., Kacorri H., Verhoef T., Vogler C., Morris M.R. Sign language recognition, generation, and translation: An interdisciplinary perspective // Proc. of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS). 2019. P. 16–31. https://doi.org/10.1145/3308561.3353774Bragg D., Koller O., Bellard M., Berke L., Boudrealt P., Braffort A., Caselli N., Huenerfauth M., Kacorri H., Verhoef T., Vogler C., Morris M.R. Sign language recognition, generation, and translation: An interdisciplinary perspective // Proc. of the 21st International ACM SIGACCESS Conference on Computers and Accessibility (ASSETS). 2019. P. 16–31. https://doi.org/10.1145/3308561.3353774Kamal S.M., Chen Y., Li S., Shi X., Zheng J. Technical approaches to Chinese sign language processing: A review // IEEE Access. 2019. V. 7. P. 96926–96935. https://doi.org/10.1109/ACCESS.2019.2929174Kamal S.M., Chen Y., Li S., Shi X., Zheng J. Technical approaches to Chinese sign language processing: A review // IEEE Access. 2019. V. 7. P. 96926–96935. https://doi.org/10.1109/ACCESS.2019.2929174O’Connor T.F., Fach M.E., Miller R., Root S.E., Mercier P.P., Lipomi D.J. The Language of Glove: Wireless gesture decoder with low-power and stretchable hybrid electronics // PLoS ONE. 2017. V. 12. N 7. P. e0179766. https://doi.org/10.1371/journal.pone.0179766O’Connor T.F., Fach M.E., Miller R., Root S.E., Mercier P.P., Lipomi D.J. The Language of Glove: Wireless gesture decoder with low-power and stretchable hybrid electronics // PLoS ONE. 2017. V. 12. N 7. P. e0179766. https://doi.org/10.1371/journal.pone.0179766Song Y., Lee S., Choi Y., Han S., Won H., Sung T.-H., Choi Y., Bae J. Design framework for a seamless smart glove using a digital knitting system // Fashion and Textiles. 2021. V. 8. N 1. P. 6. https://doi.org/10.1186/s40691-020-00237-2Song Y., Lee S., Choi Y., Han S., Won H., Sung T.-H., Choi Y., Bae J. Design framework for a seamless smart glove using a digital knitting system // Fashion and Textiles. 2021. V. 8. N 1. P. 6. https://doi.org/10.1186/s40691-020-00237-2Zhou Z., Chen K., Li X., Zhang S., Wu Y., Zhou Y., Meng K., Sun C., He Q., Fan W., Fan E., Lin Z., Tan X., Deng W., Yang J., Chen J. Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays // Nature Electronics. 2020. V. 3. N 9. P. 571–578. https://doi.org/10.1038/s41928-020-0428-6Zhou Z., Chen K., Li X., Zhang S., Wu Y., Zhou Y., Meng K., Sun C., He Q., Fan W., Fan E., Lin Z., Tan X., Deng W., Yang J., Chen J. Sign-to-speech translation using machine-learning-assisted stretchable sensor arrays // Nature Electronics. 2020. V. 3. N 9. P. 571–578. https://doi.org/10.1038/s41928-020-0428-6Bernhardt P. Myo SDK Beta 7 [Электронный ресурс]. URL: https://developerblog.myo.com/myo-sdk-beta-7/ (дата обращения: 10.02.2022).Bernhardt P. Myo SDK Beta 7 [Электронный ресурс]. URL: https://developerblog.myo.com/myo-sdk-beta-7/ (дата обращения: 10.02.2022).Abreu J.G., Teixeira J.M., Figueiredo L.S., Teichrieb V. Evaluating sign language recognition using the Myo armband // Proc. of the 18th Symposium on Virtual and Augmented Reality (SVR). 2016. P. 64–70. https://doi.org/10.1109/SVR.2016.21Abreu J.G., Teixeira J.M., Figueiredo L.S., Teichrieb V. Evaluating sign language recognition using the Myo armband // Proc. of the 18th Symposium on Virtual and Augmented Reality (SVR). 2016. P. 64–70. https://doi.org/10.1109/SVR.2016.21Wang Y., Yao Q., Kwok J., Ni L.M. Generalizing from a few examples: A survey on few-shot learning // ACM Computing Surveys. 2020. V. 53. N 3. P. 63. https://dl.acm.org/doi/10.1145/3386252Wang Y., Yao Q., Kwok J., Ni L.M. Generalizing from a few examples: A survey on few-shot learning // ACM Computing Surveys. 2020. V. 53. N 3. P. 63. https://dl.acm.org/doi/10.1145/3386252Wang F., Zhao S., Zhou X., Li C., Li M., Zeng Z. An recognition–verification mechanism for real-time Chinese sign language recognition based on multi-information fusion // Sensors. 2019. V. 19. N 11. P. 2495. https://doi.org/10.3390/s19112495Wang F., Zhao S., Zhou X., Li C., Li M., Zeng Z. An recognition–verification mechanism for real-time Chinese sign language recognition based on multi-information fusion // Sensors. 2019. V. 19. N 11. P. 2495. https://doi.org/10.3390/s19112495Kim S., Kim J., Ahn S., Kim Y. Finger language recognition based on ensemble artificial neural network learning using armband EMG sensors // Technology and Health Care. 2018. V. 26. S. 1. P. 249–258. https://doi.org/10.3233/THC-174602Kim S., Kim J., Ahn S., Kim Y. Finger language recognition based on ensemble artificial neural network learning using armband EMG sensors // Technology and Health Care. 2018. V. 26. S. 1. P. 249–258. https://doi.org/10.3233/THC-174602Paudyal P., Lee J., Banerjee A., Sandeep K.S. A comparison of techniques for sign language alphabet recognition using armband wearables // ACM Transactions on Interactive Intelligent Systems. 2019. V. 9. N 2-3. P. 1–26. https://doi.org/10.1145/3150974Paudyal P., Lee J., Banerjee A., Sandeep K.S. A comparison of techniques for sign language alphabet recognition using armband wearables // ACM Transactions on Interactive Intelligent Systems. 2019. V. 9. N 2-3. P. 1–26. https://doi.org/10.1145/3150974Tateno S., Liu H., Ou J. Development of sign language motion recognition system for hearing-impaired people using electromyography signal // Sensors. 2020. V. 20. N 20. P. 5807. https://doi.org/10.3390/s20205807Tateno S., Liu H., Ou J. Development of sign language motion recognition system for hearing-impaired people using electromyography signal // Sensors. 2020. V. 20. N 20. P. 5807. https://doi.org/10.3390/s20205807Sheng X., Lv B., Guo W., Zhu X. Common spatial-spectral analysis of EMG signals for multiday and multiuser myoelectric interface // Biomedical Signal Processing and Control. 2019. V. 53. P. 101572. https://doi.org/10.1016/j.bspc.2019.101572Sheng X., Lv B., Guo W., Zhu X. Common spatial-spectral analysis of EMG signals for multiday and multiuser myoelectric interface // Biomedical Signal Processing and Control. 2019. V. 53. P. 101572. https://doi.org/10.1016/j.bspc.2019.101572Zhang L., Shi Y., Wang W., Chu Y., Yuan X. Real-time and userindependent feature classification of forearm using EMG signals // Journal of the Society for Information Display. 2019. V. 27. N 2. P. 101–107. https://doi.org/10.1002/jsid.749Zhang L., Shi Y., Wang W., Chu Y., Yuan X. Real-time and userindependent feature classification of forearm using EMG signals // Journal of the Society for Information Display. 2019. V. 27. N 2. P. 101–107. https://doi.org/10.1002/jsid.749Das P., Paul S., Ghosh J., Palbhowmik S., Neo-Gi B., Ganguly A. An approach towards the representation of sign language by electromyography signals with fuzzy implementation // International Journal of Sensors, Wireless Communications and Control. 2017. V. 7. N 1. P. 26–32. https://doi.org/10.2174/2210327907666170222093839Das P., Paul S., Ghosh J., Palbhowmik S., Neo-Gi B., Ganguly A. An approach towards the representation of sign language by electromyography signals with fuzzy implementation // International Journal of Sensors, Wireless Communications and Control. 2017. V. 7. N 1. P. 26–32. https://doi.org/10.2174/2210327907666170222093839Côté-Allard U., Fall C.L., Drouin A., Campeau-Lecours A., Gosselin C., Glette K., Laviolette F., Gosselin B. Deep learning for electromyographic hand gesture signal classification using transfer learning // IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2019. V. 27. N 4. P. 760–771. https://doi.org/10.1109/TNSRE.2019.2896269Côté-Allard U., Fall C.L., Drouin A., Campeau-Lecours A., Gosselin C., Glette K., Laviolette F., Gosselin B. Deep learning for electromyographic hand gesture signal classification using transfer learning // IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2019. V. 27. N 4. P. 760–771. https://doi.org/10.1109/TNSRE.2019.2896269Tsinganos P., Cornelis B., Cornelis J., Jansen B., Skodras A. Data augmentation of surface electromyography for hand gesture recognition // Sensors. 2020. V. 20. N 17. P. 4892. https://doi.org/10.3390/s20174892Tsinganos P., Cornelis B., Cornelis J., Jansen B., Skodras A. Data augmentation of surface electromyography for hand gesture recognition // Sensors. 2020. V. 20. N 17. P. 4892. https://doi.org/10.3390/s20174892Li W., Shi P., Yu H. Gesture recognition using surface electromyography and deep learning for prostheses hand: state-ofthe-art, challenges, and future // Frontiers in Neuroscience. 2021. V. 15. P. 621885 https://doi.org/10.3389/fnins.2021.621885Li W., Shi P., Yu H. Gesture recognition using surface electromyography and deep learning for prostheses hand: state-ofthe-art, challenges, and future // Frontiers in Neuroscience. 2021. V. 15. P. 621885 https://doi.org/10.3389/fnins.2021.621885Rahimian E., Zabihi S., Asif A., Farina D., Atashzar S.F., Mohammadi A. FS-HGR: Few-shot learning for hand gesture recognition via electromyography // IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2021. V. 29. P. 1004–1015. https://doi.org/10.1109/TNSRE.2021.3077413Rahimian E., Zabihi S., Asif A., Farina D., Atashzar S.F., Mohammadi A. FS-HGR: Few-shot learning for hand gesture recognition via electromyography // IEEE Transactions on Neural Systems and Rehabilitation Engineering. 2021. V. 29. P. 1004–1015. https://doi.org/10.1109/TNSRE.2021.3077413Finn C., Abbeel P., Levine S. Model-agnostic meta-learning for fast adaptation of deep networks // Proceedings of Machine Learning Research. 2017. V. 70. P. 1126–1135.Finn C., Abbeel P., Levine S. Model-agnostic meta-learning for fast adaptation of deep networks // Proceedings of Machine Learning Research. 2017. V. 70. P. 1126–1135.Lee Y., Choi S. Gradient-based meta-learning with learned layerwise metric and subspace // Proc. of the 35th International Conference on Machine Learning (ICML). V. 7. 2018. P. 4574–4586.Lee Y., Choi S. Gradient-based meta-learning with learned layerwise metric and subspace // Proc. of the 35th International Conference on Machine Learning (ICML). V. 7. 2018. P. 4574–4586.Koch G., Zemel R., Salakhutdinov R. Siamese neural networks for one-shot image recognition // ICML Deep Learning Workshop. V. 2. 2015.Koch G., Zemel R., Salakhutdinov R. Siamese neural networks for one-shot image recognition // ICML Deep Learning Workshop. V. 2. 2015.Snell J., Swersky K., Zemel R. Prototypical networks for few-shot learning // Advances in Neural Information Processing Systems. 2017. P. 4077–4087.Snell J., Swersky K., Zemel R. Prototypical networks for few-shot learning // Advances in Neural Information Processing Systems. 2017. P. 4077–4087.Vaezi Joze H.R., Koller O. MS-ASL: A large-scale data set and benchmark for understanding American sign language // Proc. of the 30th British Machine Vision Conference (BMVC). 2019.Vaezi Joze H.R., Koller O. MS-ASL: A large-scale data set and benchmark for understanding American sign language // Proc. of the 30th British Machine Vision Conference (BMVC). 2019.De Coster M., Van Herreweghe M., Dambre J. Sign language recognition with transformer networks // Proc. 12th International Conference on Language Resources and Evaluation. LREC. 2020. P. 6018–6024.De Coster M., Van Herreweghe M., Dambre J. Sign language recognition with transformer networks // Proc. 12th International Conference on Language Resources and Evaluation. LREC. 2020. P. 6018–6024.Pigou L., Van Herreweghe M., Dambre J. Sign classification in sign language corpora with deep neural networks // Proc. of the International Conference on Language Resources and Evaluation (LREC), Workshop. 2016. P. 175–178.Pigou L., Van Herreweghe M., Dambre J. Sign classification in sign language corpora with deep neural networks // Proc. of the International Conference on Language Resources and Evaluation (LREC), Workshop. 2016. P. 175–178.Pradhan A., He J., Jiang N. Performance optimization of surface electromyography based biometric sensing system for both verification and identification // IEEE Sensors Journal. 2021. V. 21. N 19. P. 21718–21729. https://doi.org/10.1109/JSEN.2021.3079428Pradhan A., He J., Jiang N. Performance optimization of surface electromyography based biometric sensing system for both verification and identification // IEEE Sensors Journal. 2021. V. 21. N 19. P. 21718–21729. https://doi.org/10.1109/JSEN.2021.3079428Young A.J., Hargrove L.J., Kuike T.A. Improving myoelectric pattern recognition robustness to electrode shift by changing interelectrode distance and electrode configuration // IEEE Transactions on Biomedical Engineering, 2012, vol. 59, no. 3, pp. 645–652. https://doi.org/10.1109/TBME.2011.2177662Young A.J., Hargrove L.J., Kuike T.A. Improving myoelectric pattern recognition robustness to electrode shift by changing interelectrode distance and electrode configuration // IEEE Transactions on Biomedical Engineering, 2012, vol. 59, no. 3, pp. 645–652. https://doi.org/10.1109/TBME.2011.2177662Benatti S., Farella E., Gruppioni E., Benini L. Analysis of robust implementation of an EMG pattern recognition based control. Proc. of the Analysis of Robust Implementation of an EMG Pattern Recognition Based Control (BIOSIGNALS), 2014, pp. 45–54. https://doi.org/10.5220/0004800300450054Benatti S., Farella E., Gruppioni E., Benini L. Analysis of robust implementation of an EMG pattern recognition based control. Proc. of the Analysis of Robust Implementation of an EMG Pattern Recognition Based Control (BIOSIGNALS), 2014, pp. 45–54. https://doi.org/10.5220/0004800300450054George J.A., Neibling A., Paskett M.D., Clark G.A. Inexpensive surface electromyography sleeve with consistent electrode placement enables dexterous and stable prosthetic control through deep learning. arXiv, 2003, arXiv:2003.00070. https://doi.org/10.48550/arXiv.2003.00070George J.A., Neibling A., Paskett M.D., Clark G.A. Inexpensive surface electromyography sleeve with consistent electrode placement enables dexterous and stable prosthetic control through deep learning. arXiv, 2003, arXiv:2003.00070. https://doi.org/10.48550/arXiv.2003.00070Vinyals O., Blundell C., Lillicrap T., Kavukcuoglu K., Wierstra D. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 2016, pp. 3637–3645.Vinyals O., Blundell C., Lillicrap T., Kavukcuoglu K., Wierstra D. Matching networks for one shot learning. Advances in Neural Information Processing Systems, 2016, pp. 3637–3645.Kaczmarek P., Mańkowski T., Tomczyński J. putEMG–A surface electromyography hand gesture recognition dataset. Sensors, 2019, vol. 19, no. 16, pp. 3548. https://doi.org/10.3390/s19163548Kaczmarek P., Mańkowski T., Tomczyński J. putEMG–A surface electromyography hand gesture recognition dataset. Sensors, 2019, vol. 19, no. 16, pp. 3548. https://doi.org/10.3390/s19163548

The authors declare that there are no conflicts of interest present.