Method for generating masks on face images and systems for their recognition

The problem of masked face recognition is investigated. It is shown that real masks of various shapes, textures and colors have become a problem for state-of-the-art face recognition systems. A reason for this is the lack of the necessary real training datasets. Creation of new data based on simple methods of forming masks on face images could solve this problem. An original method is proposed including the generation of various types, shapes, and colors of masks directly on the original texture of face images. The formation of the masks on the faces of individuals, on faces in group photos, and in scenes with streams of people was taken into account. Based on 100 original face images from the CUHK Face Sketch Database, a  test database was created that includes more than 20,000 masked faces images which available for use. Experiments were carried out to recognize faces from the test database within the implemented four systems, among which three are state-of-the-art systems based on “deep learning” and one is deterministic system based on the cosine-transform. The performance of these systems was evaluated, the obtained results of masked face recognition were interpreted, and the masks that were a problem for selected four systems were noted. The proposed mask generation method can be used to create corpora and test databases of images with masks. The obtained results will be useful to researchers and specialists in the field of image processing and analysis.

1. Ge S., Li J., Ye Q., Luo Z. Detecting masked faces in the wild with LLE-CNNs. Proc. of the 30th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 426–434. https://doi.org/10.1109/CVPR.2017.53

2. Ryumina E., Ryumin D., Ivanko D., Karpov A. A novel method for protective face mask detection using convolutional neural networks and image histograms. International Archives of the Photogrammetry Remote Sensing and Spatial Information Sciences, 2021, vol. XLIV2/W1-2021, pp. 177–182. https://doi.org/10.5194/isprs-archivesXLIV-2-W1-2021-177-2021

3. Kosulina K.E., Karpov A.A., Methods for audiovisual recognition of people in masks. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2022, vol. 22, no. 3, in press. (in Russian)

4. Hsu G.-S.J., Wu H.-Y., Tsai C.-H., Yanushkevich S., Gavrilova M. Masked face recognition from synthesis to reality. IEEE Access, 2022, vol. 10, pp. 37938–37952. https://doi.org/10.1109/ACCESS.2022.3160828

5. Ngan M.L., Grother P. J., Hanaoka K. K. Ongoing Face Recognition Vendor Test (FRVT) Part 6A: Face recognition accuracy with masks using pre-COVID-19 algorithms. NISTIR 8311. National Institute of Standards and Technology U.S. Department of Commerce, 2020, 58 p. https://doi.org/10.6028/NIST.IR.8311

6. Gorodnichy D., Yanushkevich S., Shmerko V. Automated border control: Problem formalization. Proc. of the IEEE Symposium on Computational Intelligence in Biometrics and Identity, 2014, pp. 118–125. https://doi.org/10.1109/CIBIM.2014.7015452

7. Huang B., Wang Z., Wang G., Jiang K., He Z., Zou H., Zou Q. Masked face recognition datasets and validation. Proc. of the 18th IEEE/CVF International Conference on Computer Vision, 2021, pp. 1487–1491. https://doi.org/10.1109/ICCVW54120.2021.00172

8. Deng J., Guo J., An X., Zhu Z., Zafeiriou S. Masked face recognition challenge: The insightface track report. Proc. of the 18th IEEE/CVF International Conference on Computer Vision, 2021, pp. 1437–1444. https://doi.org/10.1109/ICCVW54120.2021.00165

9. Zhu Z., Huang G., Deng J., Ye Y., Huang J., Chen X., Zhu J., Yang T., Guo J., Lu J., Du D., Zhou J. Masked face recognition challenge: The webface260m track report. arXiv, 2021, arXiv.2108.07189. https://doi.org/10.48550/arXiv.2108.07189

10. Adjabi I., Ouahabi A., Benzaoui A., Taleb-Ahmed A. Past, present, and future of face recognition: A review. Electronics, 2020, vol. 9, no. 8, pp. 1188. https://doi.org/10.3390/electronics9081188

11. Alzu’bi A., Albalas F., Al-Hadhrami T., Younis L.B., Bashayreh A. Masked face recognition using deep learning: A review. Electronics, 2021, vol. 10, no. 21, pp. 2666. https://doi.org/10.3390/electronics10212666

12. Deng J., Guo J., Ververas E., Kotsia I., Zafeiriou S. RetinaFace: Single-shot multi-level face localisation in the wild. Proc. of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2 0 2 0 , p p . 5 2 0 2 – 5 2 11 . h t t p s : / / d o i . o rg / 1 0 . 11 0 9 / CVPR42600.2020.00525

13. He K., Zhang X., Ren S., Sun J. Deep residual learning for image recognition. Proc. of the 29th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016, pp. 770–778. https://doi.org/10.1109/CVPR.2016.90

14. Simonyan K., Zisserman A. Very deep convolutional networks for large-scale image recognition. Proc. of the 3rd International Conference on Learning Representations (ICLR), 2015, pp. 1–14.

15. Serengil S. I., Ozpinar A. LightFace: A hybrid deep face recognition framework. Proc. of the IEEE Innovations in Intelligent Systems and Applications Conference (ASYU), 2020, pp. 9259802. https://doi.org/10.1109/ASYU50717.2020.9259802

16. Viola P., Jones M.J. Robust real-time face detection. International Journal of Computer Vision, 2004, vol. 57, no. 2, pp. 137–154. https://doi.org/10.1023/B:VISI.0000013087.49260.fb

17. Kazemi V., Sullivan J. One millisecond face alignment with an ensemble of regression trees. Proc. of the 27th IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2014, pp. 1867–1874. https://doi.org/10.1109/CVPR.2014.241

18. Shimrat M. Algorithm 112: position of point relative to polygon. Communications of the ACM, 1962, vol. 5, no. 8, pp. 434. https://doi.org/10.1145/368637.368653

19. Ageev M.I., Alik V.P., Markov Iu.I. Algorithms 101б-150б Library. Handbook. Moscow, Radio i svjaz’ Publ., 1978, 128 p. (in Russian)

20. Grichishin Ia.T., Efimov V.I., Lomakovich A.N. Algorithms and Programs in BASIC. Moscow, Prosveshhenie Publ., 1988, 160 p. (in Russian)

21. Shchegoleva N.L., Kukharev G.A. A simple classification algorithm for linearly inseparable data. Natural and Technical Sciences, 2012, no. 1(57), pp. 358–364. (in Russian)

22. Loey M., Manogaran G., Taha M.H.N., Khalifa N.E.M. A hybrid deep transfer learning model with machine learning methods for face mask detection in the era of the COVID-19 pandemic. Measurement, 2021, vol. 167, pp. 108288. https://doi.org/10.1016/j.measurement.2020.108288

23. Kukharev G.A., Kamenskaya A.I., Matveev Y.N. Methods of facial images processing and recognition in biometrics. St.Petersburg, Politechnika Publ., 2013, 388 p. (in Russian)

24. Cao Q., Shen L., Xie W., Parkhi O.M., Zisserman A. VGGFace2: A dataset for recognising faces across pose and age. Proc. of the 13th IEEE International Conference on Automatic Face & Gesture Recognition (FG), 2018, pp. 67–74. https://doi.org/10.1109/FG.2018.00020

25. Kukharev G.A., Maulenov K.S., Shchegoleva N.L. Protecting facial images from recognition on social media: solution methods and their perspective. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2021, vol. 21, no. 5, pp. 755–766. (in Russian). https://doi.org/10.17586/2226-1494-2021-21-5-755-766