Facial keypoints detection using capsule neural networks
https://doi.org/10.17586/2226-1494-2023-23-3-506-518
Abstract
The problem of detecting key points of the face is investigated. This problem is quite relevant and important. The existing approaches of solving this problem, which are usually divided into parametric and nonparametric methods, are considered. As a result of the study, it was concluded that, nowadays, the most qualitative results are demonstrated by approaches based on deep learning methods. Two solutions are proposed: a capsule network with dynamic routing and a deep capsule network. The data for the experiments are 10,000 generated faces taken from Kaggle, marked up using MediaPipe. A method of using capsule architectures in neural networks to solve the problem of detecting key points of the face is proposed. The method includes the use of segmentation based on the key points of the face recognized using MediaPipe. Delaunay triangulation was used to build the face mesh. The architecture of a deep capsule network considering semantic segmentation was proposed. Based on the marked-up data, experiments on the detection of key points using the developed capsule neural networks were performed. According to the test results, the loss function reached values in range 2.50–2.90, the accuracy reached values in range 0.87–0.9. The proposed architecture can be used in technologies for comparing the geometry of the face grid of a real person with the geometry of the face grid of a three-dimensional model as well as in further studies of capsule neural networks by researchers in the field of image processing and analysis.
About the Authors
A. A. Boitsev
ITMO University
Russian Federation
Russian Federation
Anton A. Boitsev — PhD (Physics & Mathematics), Associate Professor
sc 56401063400
Saint Petersburg, 197101
D. G. Volchek
ITMO University
Russian Federation
Russian Federation
Dmitry G. Volchek — PhD, Associate Professor
sc 57197732532
Saint Petersburg, 197101
E. N. Magazenkov
ITMO University
Russian Federation
Russian Federation
Egor N. Magazenkov — Student
Saint Petersburg, 197101
M. K. Nevaev
ZAO “Center of Financial Technologies”
Russian Federation
Russian Federation
Maxim K. Nevaev — Systems Designer
Saint Petersburg, 191002
A. A. Romanov
ITMO University
Russian Federation
Russian Federation
Aleksei A. Romanov — PhD, Associate Professor
sc 57194976341
Saint Petersburg, 197101
References
1. Volkova S.S., Matveev Yu.N. Convolutional neural networks for face anti-spoofing. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2017, vol. 17, no. 4, pp. 702– 710. (in Russian). https://doi.org/10.17586/2226-1494-2017-17-4-702-710
2. Dikiy D.I., Artemeva V.D. Research of artificial neural network applicability for user`s online handwritten signature verification. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2017, vol. 17, no. 4, pp. 677– 684. (in Russian). https://doi.org/10.17586/2226-1494-2017-17-4-677-684
3. Abiodun O.I., Kiru M.U., Jantan A., Omolara A.E., Dada K.V., Umar A.M., Linus O.U., Arshad H., Kazaure A.A., Gana U. Comprehensive review of artificial neural network applications to pattern recognition. IEEE Access, 2019, vol. 7, pp. 158820–158846. https://doi.org/10.1109/access.2019.2945545
4. Krizhevsky A., Sutskever I., Hinton G.E. ImageNet classification with deep convolutional neural networks. Communications of the ACM, 2017, vol. 60, no. 6, pp. 84–90. https://doi.org/10.1145/3065386
5. He K., Zhang X., Ren S., Sun J. Deep residual learning for image recognition. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778. https://doi.org/10.1109/cvpr.2016.90
6. Redmon J., Divvala S., Girshick R., Farhadi A. You only look once: Unified, real-time object detection. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 779– 788. https://doi.org/10.1109/cvpr.2016.91
7. Sabour S., Frosst N., Hinton G.E. Dynamic routing between capsules. Advances in Neural Information Processing Systems, 2017, vol. 30, pp. 3856–3866.
8. Nguyen H.H., Yamagishi J., Echizen I. Capsule-forensics: Using capsule networks to detect forged images and videos. Proc. of the ICASSP 2019-2019 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), 2019, pp. 2307–2311. https://doi.org/10.1109/icassp.2019.8682602
9. Wang N., Gao X., Tao D., Yang H., Li X. Facial feature point detection: A comprehensive survey. Neurocomputing, 2018, vol. 275, pp. 50–65. https://doi.org/10.1016/j.neucom.2017.05.013
10. Beham M.P., Roomi S.M.M. A review of face recognition methods. International Journal of Pattern Recognition and Artificial Intelligence, 2013, vol. 27, no. 4, pp. 1356005. https://doi.org/10.1142/S0218001413560053
11. Li H., Lin Z.L., Brandt J., Shen X., Hua G. Efficient boosted exemplar-based face detection. Proc. of the 2014 IEEE Conference on Computer Vision and Pattern Recognition, 2014, pp. 1843–1850. https://doi.org/10.1109/cvpr.2014.238
12. Sun Y., Wang X., Tang X. Deep convolutional network cascade for facial point detection. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition, 2013, pp. 3476–3483. https://doi.org/10.1109/cvpr.2013.446
13. Zhang Z., Luo P., Loy C.C., Tang X. Facial landmark detection by deep multi-task learning. Lecture Notes in Computer Science, 2014, vol. 8694, pp. 94–108. https://doi.org/10.1007/978-3-319-10599-4_7
14. Trigeorgis G., Snape P., Nicolaou M.A., Antonakos E., Zafeiriou S. Mnemonic descent method: A recurrent process applied for end-toend face alignment. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4177–4187. https://doi.org/10.1109/cvpr.2016.453
15. Zhu X., Lei Z., Liu X., Shi H., Li S.Z. Face alignment across large poses: A 3D solution. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 146–155. https://doi.org/10.1109/cvpr.2016.23
16. Jourabloo A., Liu X. Large-pose face alignment via CNN-based dense 3D model fitting. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 4188–4196. https://doi.org/10.1109/cvpr.2016.454
17. Kartynnik Y., Ablavatski A., Grishchenko I., Grundmann M. Realtime facial surface geometry from monocular video on mobile GPUs. arXiv, 2019, arXiv:1907.06724. https://doi.org/10.48550/arXiv.1907.06724
18. Wood E., Baltrušaitis T., Hewitt Ch., Johnson M., Shen J., Milosavljević N., Wilde D., Garbin S., Sharp T., Stojiljković I., Cashman T., Valentin J. 3D face reconstruction with dense landmarks. Lecture Notes in Computer Science, 2022, vol. 13673, pp. 160–177. https://doi.org/10.1007/978-3-031-19778-9_10
19. Pechyonkin M. Understanding Hinton’s Capsule Networks. Part I: Intuition. Medium. 2018, December 18. Available at: https://medium.com/ai%C2%B3-theory-practice-business/understanding-hintonscapsule-networks-part-i-intuition-b4b559d1159b (accessed: 12.12.2022).
20. Yu D., Wang H., Chen P., Wei Z. Mixed pooling for convolutional neural networks. Lecture Notes in Computer Science, 2014, vol. 8818, pp. 364–375. https://doi.org/10.1007/978-3-319-11740-9_34
21. Ding Y., Chen F., Zhao Y., Wu Z., Zhang C., Wu D. A stacked multiconnection simple reducing net for brain tumor segmentation. IEEE Access, 2019, vol. 7, pp. 104011–104024. https://doi.org/10.1109/access.2019.2926448
Review
For citations:
Boitsev A.A., Volchek D.G., Magazenkov E.N., Nevaev M.K., Romanov A.A. Facial keypoints detection using capsule neural networks. Scientific and Technical Journal of Information Technologies, Mechanics and Optics. 2023;23(3):506-518. (In Russ.) https://doi.org/10.17586/2226-1494-2023-23-3-506-518
Views:
17
Email this article 