Algorithm for human interaction with a model of an industrial cyber-physical system by means of neural interface

   The article proposes an algorithm of a Brain Computer Interface (BCI) for implementation of interaction between a human and a model of an industrial cyberphysical system. The interface facilitates selecting a conceived tool on the basis of the classification of evoked responses of a test person’s encephalogram to visual stimuli (tool images). To conduct the study there has been designed a software system operated with a web-server, a controller, and a user BCI. The cerebral bioelectrical activity of a test person has been constantly registered with the encephalograph produced by LLC MITSAR followed by online signal processing conducted by the designed original software system. The stored evoked responses to stimuli have been classified in a variety of ways — peak-based selection, a support vector machine, and a neural net. There has been proved that accuracy of the classification of evoked potentials both with the help of a neural net and a support vector machine are approximately equal and these algorithms can be implemented in the online mode. Analysis of the experiments performed has shown that the proposed algorithm makes it possible to classify presented visual stimuli in neural interfaces in the online mode. The results show how it is possible to organize a ‘deeply integrated’ interaction between a human and an equipment through an impact of commands based on the processed signals of bioelectrical brain activity of a human on a 3D model of a production site.

1. Nazarenko A.A., Camarinha-Matos L.M. A human-al framework to design collaborative cyber physical systems. IFIP Advances in Information and Communication Technology, 2024, vol. 716, pp. 28–42. doi: 10.1007/978-3-031-63851-0_2

2. Colombo A.W., Karnouskos S., Hanisch C. Engineering human-focused Industrial Cyber-Physical Systems in Industry 4.0 context. Philosophical Transactions of the Royal Society A Mathematical Physical and Engineering Sciences, 2021, vol. 379, no. 2207, pp. 20200366. doi: 10.1098/rsta.2020.0366

3. Schmidt D., Diez J.V., Ordieres-Meré J., Gevers R., Schwiep J., Molina M. Industry 4.0 lean shopfloor management characterization using EEG sensors and deep learning. Sensors, 2020, vol. 20, no. 10, pp. 2860. doi: 10.3390/s20102860

4. Piardi L., Leitão P., Queiroz J., Pontes J. Role of digital technologies to enhance the human integration in industrial cyber–physical systems. Annual Reviews in Control, 2024, vol. 57, pp. 100934. doi: 10.1016/j.arcontrol.2024.100934

5. Zhou J., Zhou Y., Wang B., Zang J. Human–Cyber–Physical Systems (HCPSs) in the context of new-generation intelligent manufacturing. Engineering, 2019, vol. 5, no. 4, pp. 624–636. doi: 10.1016/j.eng.2019.07.015

6. Slavic D., Marjanovic U., Pezzotta G., Turcin I., Rakic S. Servitization and industry 5.0: the future trends of manufacturing transformation. IFIP Advances in Information and Communication Technology, 2023, vol. 690, pp. 109–121. doi: 10.1007/978-3-031-43666-6_8

7. Slavic D., Romero D., Pezzotta G., Marjanovic U., Savkovic B., Popan I.A., Rakic S. Towards human-centric digital services: a development framework. IFIP Advances in Information and Communication Technology, 2024, vol. 732, pp. 184–197. doi: 10.1007/978-3-031-71637-9_13

8. Reddy K.K., Reddy A.V., Doss S., Priyanka K. Human-machine collaboration and emotional intelligence in Industry 6.0: concepts, challenges, and future directions. Examining the Metaverse in Healthcare, 2024, pp. 221–246. doi: 10.4018/979-8-3693-1515-6.ch009

9. Babkin A.V., Shkarupeta E.V. Industry 6.0: the essence, trends and strategic opportunities for Russia. Russian Journal of Industrial Economics, 2024, vol.17, no. 4, pp. 353-377. (in Russian). doi: 10.17073/2072-1633-2024-4-1369

10. Värbu K., Muhammad N., Muhammad Y. Past, present, and future of EEG-based BCI applications. Sensors, 2022, vol. 2, no. 9, pp. 3331. doi: 10.3390/s22093331

11. Kitchener P.D., Hales C.G. What neuroscientists think, and don’t think, about consciousness. Frontiers in Human Neuroscience, 2022, vol. 16, pp. 767612. doi: 10.3389/fnhum.2022.767612

12. Douibi K., Le Bars S., Lemontey A., Nag L., Balp R., Breda G. Toward EEG-based BCI applications for Industry 4.0: challenges and possible applications. Frontiers in Human Neuroscience, 2021, vol. 15, pp. 705064. doi: 10.3389/fnhum.2021.705064

13. Gundelakh F., Stankevich L., Sonkin K., Nagornova G., Shemyakina N. Application of Brain-computer interfaces in assistive technologies. SPIIRAS Proceedings, vol. 19, no. 2, pp. 277–301. (in Russian). doi: 10.15622/sp.2020.19.2.2

14. Gundelakh F., Stankevich L., Sonkin K. Mobile robot control based on noninvasive brain-computer interface using hierarchical classifier of imagined motor commands. MATEC Web of Conferences, 2018, vol. 161, pp. 03003. doi: 10.1051/matecconf/201816103003

15. Kapralov N., Nagornova Z., Shemyakina N. Classification methods for EEG patterns of imaginary movements. Informatics and Automation, 2021. vol. 20, no. 1, pp. 94–132. (in Russian). doi: 10.15622/ia.2021.20.1.4

16. Guo Z., Wu J., Song Y., Bu J., Mai W., Zheng Q., Ouyang W., Song C. Neuro-3D: towards 3D visual decoding from EEG signals. arXiv, 2024, arXiv:2411.12248 doi: 10.48550/arXiv.2411.12248

17. Uchimura T. Robots or 3D models control by Brain-Computer Interfaces. Dissertation for the degree of Master of Science (Computer Science). Flinders University, Adelaide, 2021. 40 p.

18. Cao B., Niu H., Hao J., Wang G. Building EEG-based CAD object selection intention discrimination model using convolutional neural network (CNN). Advanced Engineering Informatics, 2022, vol. 52, pp. 101548. doi: 10.1016/j.aei.2022.101548

19. Duncana C.C., Barryb R.J., Connollyc J.F., Fischer C., Michie P.T., Näätänen R., et al. Event-related potentials in clinical research: Guidelines for eliciting, recording, and quantifying mismatch negativity, P300, and N400. Clinical Neurophysiology, 2009, vol. 120, no. 11, pp. 1883–1908. doi: 10.1016/j.clinph.2009.07.045

20. Gonzalez-Santocildes A., Vazquez J.-I., Eguiluz A. Enhancing robot behavior with EEG, reinforcement learning and beyond : a review of techniques in collaborative robotics. Applied Sciences, 2024, vol. 14, no. 14, pp. 6345. doi: 10.3390/app14146345

21. Aleksandrov M.V., Ivanov L.B., Lytaev S.A. Electroencephalography. St. Petersburg, SpecLit Publ., 2020, 224 p. (in Russian)

22. Marusina M.Ya., Burdaev I.V. Automatic recognition of visual stimules by single evoked potentials on an electroencephalogram. Journal of Instrument Engineering, 2023, vol. 66, no. 3, pp. 247-250. (in Russian). doi: 10.17586/0021-3454-2023-66-3-247-250

23. Baiano C., Zeppieri M. Visual Evoked Potential. StatPearls [Internet], 2023, PMID: 35881733.

24. Flach P. Machine Learning: The Art and Science of Algorithms that Make Sense of Data. Cambridge University Press, 2012, 409 p.

25. Weidman S. Deep Learning from Scratch: Building with Python from First Principles. O’Reilly Media, 2019, 250 p.

26. Qi J., Hu J., Peng Y., Ren Q., Wang W., Zhan Z. Integration of similarity measurement and dynamic SVM for electrically evoked potentials prediction in visual prostheses research. Expert Systems with Applications, 2011, vol. 38, no. 5, pp. 5044–5060. doi: 10.1016/j.eswa.2010.09.145

27. Swihart S.L., Matheny A. Classification of chromatic visual evoked potentials with the aid of a neural net. Computers in Biology and Medicine, 1992, vol. 22, no. 3, pp. 165–171. doi: 10.1016/0010-4825(92)90012-c

28. Zheng X., Cao Z., Bai Q. An evoked potential-guided deep learning brain representation for visual classification. Communications in Computer and Information Science, 2020, vol. 1333, pp. 54–61. doi: 10.1007/978-3-030-63823-8_7

29. Zhang Y., Xie S., Wang H., Zhang Z. Data analytics in steady-state visual evoked potential-based brain-computer interface : a review. IEEE Sensors Journal, 2021, vol. 21, no. 2, pp. 1124–1138. doi: 10.1109/jsen.2020.3017491

30. Cui Z., Chen W., Chen Y. Multi-scale convolutional neural networks for time series classification. arXiv, 2016, arXiv:1603.06995. https://arxiv.org/abs/1603.06995