ntvНаучно-технический вестник информационных технологий, механики и оптикиScientific and Technical Journal of Information Technologies, Mechanics and Optics2226-14942500-0373Университет ИТМО10.17586/2226-1494-2023-23-6-1178-1186ntv-32Research ArticleИСКУССТВЕННЫЙ ИНТЕЛЛЕКТ И КОГНИТИВНЫЕ ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИARTIFICIAL INTELLIGENCE AND COGNITIVE INFORMATION TECHNOLOGIESПерсонализация сверточных нейронных сетей в задаче обнаружения стресса с использованием данных вариабельности сердечного ритмаPersonalization of convolutional neural networks within the stress detection task using heart rate variability datahttps://orcid.org/0000-0002-0571-5836ДоброхваловМ. О.DobrokhvalovM. O.

Доброхвалов Максим Олегович — аспирант

Санкт-Петербург, 197022

Maksim O. Dobrokhvalov — PhD Student

Saint Petersburg, 197022

night1337bot@gmail.comhttps://orcid.org/0000-0003-4298-8523ФилатовА. Ю.FilatovA. Yu.

Филатов Антон Юрьевич — кандидат технических наук, доцент

Санкт-Петербург, 197022

sc 57194078312

Anton Yu. Filatov — PhD, Associate Professor

Saint Petersburg, 197022

sc 57194078312

aifilatov@etu.ruСанкт-Петербургский государственный электротехнический университет «ЛЭТИ» имени В.И. Ульянова (Ленина)РоссияSaint Petersburg Electrotechnical University “LETI”Russian Federation20231112202423611781186Copyright © Доброхвалов М.О., Филатов А.Ю., 20242024Доброхвалов М.О., Филатов А.Ю.Dobrokhvalov M.O., Filatov A.Y.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ntv.elpub.ru/jour/article/view/32

Введение. Обнаружение стресса является активной областью исследований с важными последствиями для личного, профессионального и социального здоровья человека. Большинство современных подходов используют признаки, вычисленные на основе нескольких сенсорных модальностей, т. е. группируют для обработки различные типы данных, полученные из нескольких источников. К ним относятся электрокардиограмма, кожно-гальваническая реакция, электромиограмма, температура кожи, дыхание, данные акселерометров и др. При этом чаще используются традиционные алгоритмы машинного обучения, такие как решающие деревья, дискриминантный анализ, метод опорных векторов и другие, а также полносвязные нейронные сети. Использование этих методов требует больших объемов данных. Исследователи рассматривают отличающиеся подходы к персонализации или общности моделей относительно субъектов, а именно субъекто-независимые и субъекто-зависимые (изначально персональные или адаптированные) модели. Целью представленной работы является разработка метода детектирования стресса на основе данных вариабельности сердечного ритма с учетом процесса персонализации нейронных сетей. Метод. Для решения поставленной задачи предложено применение сверточной нейронной сети. Исследована зависимость точности детектирования от длины входного сигнала. Рассмотрена зависимость точности от используемого в сети слоя уменьшения размерности данных (одномерный сверточный слой, максимизирующий и усредняющий пуллинги). Продемонстрирована важность персонализации моделей, для значительного увеличения точности детектирования для конкретных субъектов. Основные результаты. Показано, что предлагаемый метод на основании 60 интервалов между ударами сердца позволяет бинарно определить, находится ли человек в состоянии стресса. Персонализация сверточных нейронных сетей позволила повысить точность с 91,8 до 98,9 ± 2,6 %. Значение F1-меры повысилось с 0,907 до 0,983 ± 0,038. Обсуждение. Предложенные персонализированные сети могут применяться в системах мониторинга функционального состояния человека. Также могут быть использованы как часть системы, предоставляющей или ограничивающей доступ к приватным ресурсам на основании того, находится ли человек в состоянии покоя в данный момент.

Stress detection is an active area of research with important implications for personal, occupational, and social health. Most modern approaches use features computed from multiple sensor modalities, i.e., grouping different types of data from multiple sources for processing. These include electrocardiogram, electrodermal activity, electromyogram, skin temperature, respiration, accelerometer data, etc. Also, traditional machine learning algorithms (decision tree, discriminant analysis, support vector machine, etc.) or fully-connected neural networks are mostly used. Using these methods requires large amounts of data. Researchers are considering different approaches to personalization or generalization of models relative to subjects, namely subject-independent and subject-dependent (initially personal or adapted) models. The aim of the presented work is to develop a method for detecting stress based on heart rate variability data, taking into account the process of personalization of neural networks. The use of a convolutional neural network is proposed. The dependence of accuracy on the length of the input signal is studied. The dependence of accuracy on the data dimensionality reduction layer (one-dimensional convolutional layer, maximizing and averaging pooling) used in the network is also considered. The importance of personalizing models is demonstrated to significantly increase the accuracy of models of specific subjects. It is shown that the proposed method, based on 60 intervals between heartbeats, makes it possible to binary determine whether a person is under stress. Personalization allowed increasing the accuracy from 91.8 % to 98.9 ± 2.6 %. The F1-score value increased from 0.907 to 0.983 ± 0.038. The proposed personalized networks can be used in systems for monitoring the functional state of a person. They can also be used as part of a system that grants or restricts access to private resources based on whether a person is currently at rest.

детектирование стрессасверточные нейронные сетимашинное обучениевариабельность сердечного ритмасубъекто-зависимые моделиstress detectionconvolutional neural networkmachine learningheart rate variabilitysubject-dependent modelsРабота подготовлена в рамках проекта «Методы гибридного интеллекта для построения гетерогенных многоагентных систем с самообучением и самоорганизацией» программы развития СПбГЭТУ «ЛЭТИ».The article was prepared within the project “Methods of hybrid intelligence for building heterogeneous multi-agent systems with self-learning and self-organization” of the development program of St. Petersburg Electrotechnical University “LETI”.ReferencesSelye H. The Stress of Life. 1956. New York, McGraw-Hill, 324 p.Selye H. The Stress of Life. 1956. New York: McGraw-Hill, 324 p.Schmidt P., Reiss A., Duerichen R., Marberger C., Van Laerhoven K. Introducing WESAD, a multimodal dataset for wearable stress and affect detection. Proc. of the 20th ACM International Conference on Multimodal Interaction, 2018, pp. 400–408. https://doi.org/10.1145/3242969.3242985Schmidt P., Reiss A., Duerichen R., Marberger C., Van Laerhoven K. Introducing WESAD, a multimodal dataset for wearable stress and affect detection // Proc. of the 20th ACM International Conference on Multimodal Interaction. 2018. P. 400–408. https://doi.org/10.1145/3242969.3242985Bobade P., Vani M. Stress detection with machine learning and deep learning using multimodal physiological data. Proc. of the Second International Conference on Inventive Research in Computing Applications (ICIRCA), 2020, pp. 51–57. https://doi.org/10.1109/icirca48905.2020.9183244Bobade P., Vani M. Stress detection with machine learning and deep learning using multimodal physiological data // Proc. of the Second International Conference on Inventive Research in Computing Applications (ICIRCA). 2020. P. 51–57. https://doi.org/10.1109/icirca48905.2020.9183244Aqajari S.A.H., Naeini E.K., Mehrabadi M.A., Labbaf S., Rahmani A.M., Dutt N. GSR analysis for stress: Development and validation of an open source tool for noisy naturalistic GSR data. arXiv, 2020, arXiv:2005.01834. https://doi.org/10.48550/arXiv.2005.01834Aqajari S.A.H., Naeini E.K., Mehrabadi M.A., Labbaf S., Rahmani A.M., Dutt N. GSR analysis for stress: Development and validation of an open source tool for noisy naturalistic GSR data // arXiv. 2020. arXiv:2005.01834. https://doi.org/10.48550/arXiv.2005.01834Sah R.K., Ghasemzadeh H. Stress classification and personalization: Getting the most out of the least. arXiv, 2021, arXiv:2107.05666. https://doi.org/10.48550/arXiv.2107.05666Sah R.K., Ghasemzadeh H. Stress classification and personalization: Getting the most out of the least // arXiv. 2021. arXiv:2107.05666. https://doi.org/10.48550/arXiv.2107.05666Ghosh S., Kim S., Ijaz M.F., Singh P.K., Mahmud M. Classification of mental stress from wearable physiological sensors using imageencoding-based deep neural network. Biosensors, 2022, vol. 12, no. 12, pp. 1153. https://doi.org/10.3390/bios12121153Ghosh S., Kim S., Ijaz M.F., Singh P.K., Mahmud M. Classification of mental stress from wearable physiological sensors using imageencoding-based deep neural network // Biosensors. 2022. V. 12. N 12. P. 1153. https://doi.org/10.3390/bios12121153Castaldo R., Xu W., Melillo P., Pecchia L., Santamaria L., James C. Detection of mental stress due to oral academic examination via ultrashort-term HRV analysis. Proc. of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2016, pp. 3805–3808. https://doi.org/10.1109/embc.2016.7591557Castaldo R., Xu W., Melillo P., Pecchia L., Santamaria L., James C. Detection of mental stress due to oral academic examination via ultrashort-term HRV analysis // Proc. of the 38th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). 2016. P. 3805–3808. https://doi.org/10.1109/embc.2016.7591557Dalal S., Khalaf O.I. Prediction of occupation stress by implementing convolutional neural network techniques. Journal of Cases on Information Technology (JCIT), 2021, vol. 23, no. 3, pp. 27–42. https://doi.org/10.4018/jcit.20210701.oa3Dalal S., Khalaf O.I. Prediction of occupation stress by implementing convolutional neural network techniques // Journal of Cases on Information Technology (JCIT). 2021. V. 23. N 3. P. 27–42. https://doi.org/10.4018/jcit.20210701.oa3Lai K., Yanushkevich S.N., Shmerko V.P. Intelligent stress monitoring assistant for first responders. IEEE Access, 2021, vol. 9, pp. 25314– 25329. https://doi.org/10.1109/access.2021.3057578Lai K., Yanushkevich S.N., Shmerko V.P. Intelligent stress monitoring assistant for first responders // IEEE Access. 2021. V. 9. P. 25314– 25329. https://doi.org/10.1109/access.2021.3057578Cho H.M., Park H., Dong S.Y., Youn I. Ambulatory and laboratory stress detection based on raw electrocardiogram signals using a convolutional neural network. Sensors, 2019, vol. 19, no. 20, pp. 4408. https://doi.org/10.3390/s19204408Cho H.M., Park H., Dong S.Y., Youn I. Ambulatory and laboratory stress detection based on raw electrocardiogram signals using a convolutional neural network // Sensors. 2019. V. 19. N 20. P. 4408. https://doi.org/10.3390/s19204408Iqbal T., Redon-Lurbe P., Simpkin A.J., Elahi A., Ganly S., Wijns W., Shahzad A. A sensitivity analysis of biophysiological responses of stress for wearable sensors in connected health. IEEE Access, 2021, vol. 9, pp. 93567–93579. https://doi.org/10.1109/access.2021.3082423Iqbal T., Redon-Lurbe P., Simpkin A.J., Elahi A., Ganly S., Wijns W., Shahzad A. A sensitivity analysis of biophysiological responses of stress for wearable sensors in connected health // IEEE Access. 2021. V. 9. P. 93567–93579. https://doi.org/10.1109/access.2021.3082423Dalmeida K.M., Masala G.L. HRV features as viable physiological markers for stress detection using wearable devices. Sensors, 2021, vol. 21, no. 8, pp. 2873. https://doi.org/10.3390/s21082873Dalmeida K.M., Masala G.L. HRV features as viable physiological markers for stress detection using wearable devices // Sensors. 2021. V. 21. N 8. P. 2873. https://doi.org/10.3390/s21082873He J., Li K., Liao X., Zhang P., Jiang N. Real-time detection of acute cognitive stress using a convolutional neural network from electrocardiographic signal. IEEE Access, 2019, vol. 7, pp. 42710– 42717. https://doi.org/10.1109/access.2019.2907076He J., Li K., Liao X., Zhang P., Jiang N. Real-time detection of acute cognitive stress using a convolutional neural network from electrocardiographic signal // IEEE Access. 2019. V. 7. P. 42710– 42717. https://doi.org/10.1109/access.2019.2907076Woodward K., Kanjo E., Brown D.J., McGinnity T.M. On-device transfer learning for personalising psychological stress modelling using a convolutional neural network. arXiv, 2020, arXiv:2004.01603. https://doi.org/10.48550/arXiv.2004.01603Woodward K., Kanjo E., Brown D.J., McGinnity T.M. On-device transfer learning for personalising psychological stress modelling using a convolutional neural network // arXiv. 2020. arXiv:2004.01603. https://doi.org/10.48550/arXiv.2004.01603Liapis A., Faliagka E., Katsanos C., Antonopoulos C., Voros N. Detection of subtle stress episodes during UX evaluation: Assessing the performance of the WESAD bio-signals dataset. Lecture Notes in Computer Science, 2021, vol. 12934, pp. 238–247. https://doi.org/10.1007/978-3-030-85613-7_17Liapis A., Faliagka E., Katsanos C., Antonopoulos C., Voros N. Detection of subtle stress episodes during UX evaluation: Assessing the performance of the WESAD bio-signals dataset // Lecture Notes in Computer Science. 2021. V. 12934. P. 238–247. https://doi.org/10.1007/978-3-030-85613-7_17Al Abdi R.M., Alhitary A.E., Abdul Hay E.W., Al-Bashir A.K. Objective detection of chronic stress using physiological parameters. Medical & Biological Engineering & Computing, 2018, vol. 56, no. 12, pp. 2273–2286. https://doi.org/10.1007/s11517-018-1854-8Al Abdi R.M., Alhitary A.E., Abdul Hay E.W., Al-Bashir A.K. Objective detection of chronic stress using physiological parameters // Medical & Biological Engineering & Computing. 2018. V. 56. N 12. P. 2273–2286. https://doi.org/10.1007/s11517-018-1854-8Indikawati F.I., Winiarti S. Stress detection from multimodal wearable sensor data. IOP Conference Series: Materials Science and Engineering, 2020, vol. 771, no. 1, pp. 012028. https://doi.org/10.1088/1757-899x/771/1/012028Indikawati F.I., Winiarti S. Stress detection from multimodal wearable sensor data // IOP Conference Series: Materials Science and Engineering. 2020. V. 771. N 1. P. 012028. https://doi.org/10.1088/1757-899x/771/1/012028Hantono B.S., Nugroho L.E., Santosa P.I. Mental stress detection via heart rate variability using machine learning. International Journal on Electrical Engineering and Informatics, 2020, vol. 12, no. 3, pp. 431–444. https://doi.org/10.15676/ijeei.2020.12.3.3Hantono B.S., Nugroho L.E., Santosa P.I. Mental stress detection via heart rate variability using machine learning // International Journal on Electrical Engineering and Informatics. 2020. V. 12. N 3. P. 431– 444. https://doi.org/10.15676/ijeei.2020.12.3.3Ninh V.T., Smyth S., Tran M.T., Gurrin C. Analysing the performance of stress detection models on consumer-grade wearable devices. Frontiers in Artificial Intelligence and Applications, 2021, vol. 337, pp. 524–537. https://doi.org/10.3233/faia210050Ninh V.T., Smyth S., Tran M.T., Gurrin C. Analysing the performance of stress detection models on consumer-grade wearable devices // Frontiers in Artificial Intelligence and Applications. 2021. V. 337. P. 524–537. https://doi.org/10.3233/faia210050Albaladejo-González M., Ruipérez-Valiente J.A., Gómez Mármol F. Evaluating different configurations of machine learning models and their transfer learning capabilities for stress detection using heart rate. Journal of Ambient Intelligence and Humanized Computing, 2023, vol. 14, no. 8, pp. 11011–11021. https://doi.org/10.1007/s12652-022- 04365-zAlbaladejo-González M., Ruipérez-Valiente J.A., Gómez Mármol F. Evaluating different configurations of machine learning models and their transfer learning capabilities for stress detection using heart rate // Journal of Ambient Intelligence and Humanized Computing. 2023. V. 14. N 8. P. 11011–11021. https://doi.org/10.1007/s12652-022-04365-zSiirtola P. Continuous stress detection using the sensors of commercial smartwatch. Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers, 2019, pp. 1 198–1201. https://doi.org/10.1145/3341162.3344831Siirtola P. Continuous stress detection using the sensors of commercial smartwatch // Adjunct Proceedings of the 2019 ACM International Joint Conference on Pervasive and Ubiquitous Computing and Proceedings of the 2019 ACM International Symposium on Wearable Computers. 2019. P. 1 1 98–1201. https://doi.org/10.1145/3341162.3344831Quinlan J.R. C4.5: Programs for Machine Learning. Elsevier, 2014, 312 p.Quinlan J.R. C4.5: Programs for Machine Learning. Elsevier, 2014. 312 p. 23. Oskooei A., Chau S.M., Weiss J., Sridhar A., Martínez M.R., Michel B. DeStress: deep learning for unsupervised identification of mental stress in firefighters from heart-rate variability (HRV) data // Studies in Computational Intelligence. 2021. V. 194. P. 93–105. https://doi.org/10.1007/978-3-030-53352-6_9Oskooei A., Chau S.M., Weiss J., Sridhar A., Martínez M.R., Michel B. DeStress: deep learning for unsupervised identification of mental stress in firefighters from heart-rate variability (HRV) data. Studies in Computational Intelligence, 2021, vol. 194, pp. 93–105. https://doi.org/10.1007/978-3-030-53352-6_9Hochreiter S., Schmidhuber J. Long short-term memory // Neural Computation. 1997. V. 9. N 8. P. 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735Hochreiter S., Schmidhuber J. Long short-term memory. Neural Computation, 1997, vol. 9, no. 8, pp. 1735–1780. https://doi.org/10.1162/neco.1997.9.8.1735Xu Q., Nwe T.L., Guan C. Cluster-based analysis for personalized stress evaluation using physiological signals // IEEE Journal of Biomedical and Health Informatics. 2015. V. 19. N 1. P. 275–281. https://doi.org/10.1109/jbhi.2014.2311044Xu Q., Nwe T.L., Guan C. Cluster-based analysis for personalized stress evaluation using physiological signals. IEEE Journal of Biomedical and Health Informatics, 2015, vol. 19, no. 1, pp. 275–281. https://doi.org/10.1109/jbhi.2014.2311044Garcia-Ceja E., Brena R. Building personalized activity recognition models with scarce labeled data based on class similarities // Lecture Notes in Computer Science. 2015. V. 9454. P. 265–276. https://doi. org/10.1007/978-3-319-26401-1_25Garcia-Ceja E., Brena R. Building personalized activity recognition models with scarce labeled data based on class similarities. Lecture Notes in Computer Science, 2015, vol. 9454, pp. 265–276. https://doi.org/10.1007/978-3-319-26401-1_25Lu H., Frauendorfer D., Rabbi M., Mast M.S., Chittaranjan G.T., Campbell A.T., Gatica-Perez D., Choudhury T. Stresssense: Detecting stress in unconstrained acoustic environments using smartphones // Proc. of the 2012 ACM Conference on Ubiquitous Computing. 2012. P. 351–360. https://doi.org/10.1145/2370216.2370270Lu H., Frauendorfer D., Rabbi M., Mast M.S., Chittaranjan G.T., Campbell A.T., Gatica-Perez D., Choudhury T. Stresssense: Detecting stress in unconstrained acoustic environments using smartphones. Proc. of the 2012 ACM Conference on Ubiquitous Computing, 2012, pp. 351–360. https://doi.org/10.1145/2370216.2370270Aladin A.I., Whelton S.P., Al-Mallah M.H., Blaha M.J., Keteyian S.J., Juraschek S.P., Rubin J., Brawner C.A., Michos E.D. Relation of resting heart rate to risk for all-cause mortality by gender after considering exercise capacity (the Henry Ford exercise testing project) // The American Journal of Cardiology. 2014. V. 114. N 11. P. 1701–1706. https://doi.org/10.1016/j.amjcard.2014.08.042Aladin A.I., Whelton S.P., Al-Mallah M.H., Blaha M.J., Keteyian S.J., Juraschek S.P., Rubin J., Brawner C.A., Michos E.D. Relation of resting heart rate to risk for all-cause mortality by gender after considering exercise capacity (the Henry Ford exercise testing project). The American Journal of Cardiology, 2014, vol. 114, no. 11, pp. 1701–1706. https://doi.org/10.1016/j.amjcard.2014.08.042Cawley G.C., Talbot N.L.C. Efficient leave-one-out cross-validation of kernel fisher discriminant classifiers // Pattern Recognition. 2003. V. 36. N 11. P. 2585–2592. https://doi.org/10.1016/s0031-3203(03)00136-5Cawley G.C., Talbot N.L.C. Efficient leave-one-out cross-validation of kernel fisher discriminant classifiers. Pattern Recognition, 2003, vol. 36, no. 11, pp. 2585–2592. https://doi.org/10.1016/s0031-3203(03)00136-5Cawley G.C., Talbot N.L.C. Efficient leave-one-out cross-validation of kernel fisher discriminant classifiers. Pattern Recognition, 2003, vol. 36, no. 11, pp. 2585–2592. https://doi.org/10.1016/s0031-3203(03)00136-5

The authors declare that there are no conflicts of interest present.