ntvНаучно-технический вестник информационных технологий, механики и оптикиScientific and Technical Journal of Information Technologies, Mechanics and Optics2226-14942500-0373Университет ИТМО10.17586/2226-1494-2025-25-4-651-662ntv-490Research ArticleКОМПЬЮТЕРНЫЕ СИСТЕМЫ И ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИCOMPUTER SCIENCEМетод генерации анимации цифрового аватара с речевой и невербальной синхронизацией на основе бимодальных данныхA method for generating digital avatar animation with speech and non-verbal synchronization based on bimodal datahttps://orcid.org/0000-0002-7479-2851АксёновА. А.AxyonovA. A.

Александр Александрович Аксёнов, кандидат технических наук, старший научный сотрудник

199178; Санкт-Петербург

sc 57203963345

Alexander A. Axyonov,  PhD, Senior Researcher

199178; Saint Petersburg

sc 57203963345

axyonov.a@iias.spb.suhttps://orcid.org/0000-0002-4135-6949РюминаЕ. В.RyuminaE. V.

Елена Витальевна Рюмина, младший научный сотрудник

199178; Санкт-Петербург

sc 57220572427

Elena V. Ryumina, Junior Researcher

199178; Saint Petersburg

sc 57220572427

ryumina.e@iias.spb.suhttps://orcid.org/0000-0002-7935-0569РюминД. А.RyuminD. A.

Дмитрий Александрович Рюмин, кандидат технических наук, старший научный сотрудник

199178; Санкт-Петербург

sc 57191960214

Dmitry A. Ryumin, PhD, Senior Researcher

199178; Saint Petersburg

sc 57191960214

ryumin.d@iias.spb.suСанкт-Петербургский Федеральный исследовательский центр Российской академии наукРоссияSt. Petersburg Federal Research Center of the Russian Academy of SciencesRussian Federation202529082025254651662Copyright © Аксёнов А.А., Рюмина Е.В., Рюмин Д.А., 20252025Аксёнов А.А., Рюмина Е.В., Рюмин Д.А.Axyonov A.A., Ryumina E.V., Ryumin D.A.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ntv.elpub.ru/jour/article/view/490   Введение

   Введение. Рассмотрена задача генерации анимации цифрового аватара с синхронным воспроизведением речи, мимики и жестикуляции на основе бимодального входа — статического изображения и текста с эмоциональной окраской. Исследована возможность интеграции акустических, визуальных и аффективных признаков в единую модель, обеспечивающую реалистичное и выразительное поведение аватара в соответствии с содержанием и эмоциональным тоном высказывания.

   Метод

   Метод. Предложенный метод включает шаги извлечения визуальных ориентиров лица, рук и позы, определения пола для выбора подходящего голосового профиля, анализа текста на предмет эмоционального содержания и генерации синтетической аудиоречи. Все признаки интегрируются в генеративной архитектуре на основе диффузионной модели с механизмами временного внимания и межмодального согласования. Это обеспечивает высокую точность синхронизации между речью и невербальными компонентами поведения аватара. Для обучения использовались два специализированных корпуса: один для моделирования жестикуляции, другой — для мимики. Аннотирование производилось средствами автоматического извлечения пространственных ориентиров.

   Основные результаты

   Основные результаты. Экспериментальное исследование метода выполнялось на многопроцессорной вычислительной платформе с графическими ускорителями. Качество работы модели оценивалось с помощью объективных метрик. Метод показал высокую степень визуального и семантического соответствия: FID — 50,13; FVD — 601,70; SSIM — 0,752; PSNR — 21,997; E-FID — 2,226; Sync-D — 7,003; Sync-C — 6,398. Модель успешно синхронизирует речь с мимикой и жестами, учитывает эмоциональный контекст текста, а также особенности русского жестового языка.

   Обсуждение

   Обсуждение. Результаты работы могут найти применение в системах эмоционально-чувствительного человеко-машинного взаимодействия, цифровых ассистентах, образовательных и психологических интерфейсах. Предложенный метод представляет интерес для специалистов в области искусственного интеллекта, мультимодальных интерфейсов, компьютерной графики и цифровой психологии.

   This paper addresses the task of generating animations of a digital avatar that synchronously reproduces speech, facial expressions, and gestures based on a bimodal input — namely, a static image and an emotionally colored text. The study explores the integration of acoustic, visual, and affective features into a unified model that enables realistic and expressive avatar behavior aligned with both the semantic content and emotional tone of the utterance. The proposed method includes several stages: extraction of visual landmarks of the face, hands, and body pose; gender recognition for selecting an appropriate voice profile; emotional analysis of the input text; and generation of synthetic speech. All extracted features are integrated within a generative architecture based on a diffusion model enhanced with temporal attention mechanisms and cross-modal alignment strategies. This ensures high-precision synchronization between speech and the avatar nonverbal behavior. The training process utilized two specialized datasets: one focused on gesture modeling, and the other on facial expression synthesis. Annotation was performed using automated spatial landmark extraction tools. Experimental evaluation was conducted on a multiprocessor computing platform with GPU acceleration. The model performance was assessed using a set of objective metrics. The proposed method demonstrated a high degree of visual and semantic coherence: FID — 50.13, FVD — 601.70, SSIM — 0.752, PSNR — 21.997, E-FID — 2.226, Sync-D — 7.003, Sync-C — 6.398. The model effectively synchronizes speech with facial expressions and gestures, accounts for the emotional context of the text, and incorporates features of Russian Sign Language. The proposed approach has potential applications in emotionally aware human — computer interaction systems, digital assistants, educational platforms, and psychological interfaces. The method is of interest to researchers in artificial intelligence, multimodal interfaces, computer graphics, and digital psychology.

цифровой аватарBiMoDiCAмимикажестылатентное пространствогенерация анимациисинтез речиDenoising U-NetStable Diffusiondigital avatarBiMoDiCAfacial expressiongestureslatent spaceanimation generationspeech synthesisDenoising U-NetStable DiffusionРазделы «Исследования в области генерации цифровых аватаров» и «Экспериментальные исследования метода» выполнены при финансовой поддержке Российского научного фонда (проект № 24-71-00083), остальные исследования — при поддержке того же фонда (проект № 24-71-00112)The “Research on Digital Avatar Generation” and “Experimental Research of the Method” sections were supported by the Russian Science Foundation (Project No. 24-71-00083), while the remaining research was supported by the same Foundation (Project No. 24-71-00112)ReferencesSincan O.M., Keles H.Y. AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods // IEEE Access. 2020. V. 8 . P. 181340–181355. doi: 10.1109/ACCESS.2020.3028072Sincan O.M., Keles H.Y. AUTSL: A Large Scale Multi-Modal Turkish Sign Language Dataset and Baseline Methods. IEEE Access, 2020, vol. 8, pp. 181340–181355. doi: 10.1109/ACCESS.2020.3028072Kapitanov A., Kvanchiani K., Nagaev A., Kraynov R., Makhliarchuk A. H aGRID-HAnd Gesture Recognition Image Dataset // Proc. of the Winter Conference on Applications of Computer Vision (WACV). 2024. P. 4560–4569. doi: 10.1109/WACV57701.2024.00451Kapitanov A., Kvanchiani K., Nagaev A., Kraynov R., Makhliarchuk A. HaGRID-HAnd Gesture Recognition Image Dataset. Proc. of the Winter Conference on Applications of Computer Vision (WACV). 2024 , pp. 4560–4569. doi: 10.1109/WACV57701.2024.00451Busso C., Bulut M., Lee C.C., Kazemzadeh A., Mower E., Kim S., Chang J., Lee S., Narayanan S.S. IEMOCAP: interactive emotional dyadic motion capture database // Language Resources and Evaluation. 2008. V. 42. N 4. P. 335–359. doi: 10.1007/s10579-008-9076-6Busso C., Bulut M., Lee C.C., Kazemzadeh A., Mower E., Kim S., Chang J., Lee S., Narayanan S.S. IEMOCAP: interactive emotional dyadic motion capture database. Language Resources and Evaluation, 2008, vol. 42, no. 4, pp. 335–359. doi: 10.1007/s10579-008-9076-6Shen K., Guo C., Kaufmann M., Zarate J., Valentin J., Song J., Hilliges O. X-Avatar: expressive human avatars // Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2023. P. 16911–16921. doi: 10.1109/CVPR52729.2023.01622Shen K., Guo C., Kaufmann M., Zarate J., Valentin J., Song J., Hilliges O. X-Avatar: expressive human avatars. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2023, pp. 16911–16921. doi: 10.1109/CVPR52729.2023.01622Zhang H., Chen B., Yang H., Qu L., Wang X., Chen L., Long C., Zhu F., Du D., Zheng M. AvatarVerse: high-quality and stable 3D avatar creation from text and pose // Proc. of the AAAI Conference on Artificial Intelligence. 2024. V. 38. N 7. P. 7124–7132. doi: 10.1609/aaai.v38i7.28540Zhang H., Chen B., Yang H., Qu L., Wang X., Chen L., Long C., Zhu F., Du D., Zheng M. AvatarVerse: high-quality and stable 3D avatar creation from text and pose. Proc. of the AAAI Conference on Artificial Intelligence, 2024, vol. 38, no. 7, pp. 7124–7132. doi: 10.1609/aaai.v38i7.28540Kim K., Song B. Robust 3D human avatar reconstruction from monocular videos using depth optimization and camera pose estimation // IEEE Access. 2025. V. 13. P. 57886–57897. doi: 10.1109/ACCESS.2025.3556445Kim K., Song B. Robust 3D human avatar reconstruction from monocular videos using depth optimization and camera pose estimation. IEEE Access, 2025, vol. 13, pp. 57886–57897. doi: 10.1109/ACCESS.2025.3556445Yuan Y., Li X., Huang Y., De Mello S., Nagano K., Kautz J., Iqbal U. Gavatar: animatable 3D gaussian avatars with implicit mesh learning // Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024. P. 896–905. doi: 10.1109/CVPR52733.2024.00091Yuan Y., Li X., Huang Y., De Mello S., Nagano K., Kautz J., Iqbal U. Gavatar: animatable 3D gaussian avatars with implicit mesh learning. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 896–905. doi: 10.1109/CVPR52733.2024.00091Teotia K., Mallikarjun B.R., Pan X., Kim H., Garrido P., Elgharib M., Theobalt C. HQ3DAvatar: high-quality implicit 3D head avatar // ACM Transactions on Graphics. 2024. V. 43. N 3. P 1–24. doi: 10.1145/3649889Teotia K., Mallikarjun B.R., Pan X., Kim H., Garrido P., Elgharib M., Theobalt C. HQ3DAvatar: high-quality implicit 3D head avatar. ACM Transactions on Graphics, 2024, vol. 43, no. 3, pp 1–24. doi: 10.1145/3649889Yang L., Zhang Z., Song Y., Hong S., Xu R., Zhao Y., Zhang W., Cui B., Yang M. Diffusion models: a comprehensive survey of methods and applications // ACM Computing Surveys. 2023. V. 56. N 4. P. 1–39. doi: 10.1145/3626235Yang L., Zhang Z., Song Y., Hong S., Xu R., Zhao Y., Zhang W., Cui B., Yang M. Diffusion models: a comprehensive survey of methods and applications. ACM Computing Surveys, 2023, vol. 56, no. 4, pp. 1–39. doi: 10.1145/3626235Karras J., Holynski A., Wang T., Kemelmacher-Shlizerman I. DreamPose: fashion image-to-video synthesis via stable diffusion // Proc. of the IEEE/CVF International Conference on Computer Vision (ICCV). 2023. P. 22623–22633. doi: 10.1109/ICCV51070.2023.02073Karras J., Holynski A., Wang T., Kemelmacher-Shlizerman I. DreamPose: fashion image-to-video synthesis via stable diffusion. Proc. of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 22623–22633. doi: 10.1109/ICCV51070.2023.02073Huang Z., Tang F., Zhang Y., Cun X., Cao J., Li J., Lee T. Make-Your-Anchor: a diffusion-based 2D avatar generation framework // Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024. P. 6997–7006. doi: 10.1109/CVPR52733.2024.00668Huang Z., Tang F., Zhang Y., Cun X., Cao J., Li J., Lee T. Make-Your-Anchor: a diffusion-based 2D avatar generation framework. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 6997–7006. doi: 10.1109/CVPR52733.2024.00668Blattmann A., Dockhorn T., Kulal S., Mendelevitch D., Kilian M., Lorenz D., Levi Y., English Z., Voleti V., Letts A., Jampani V., Rombach R. Stable video diffusion: scaling latent video diffusion models to large datasets // arXiv. 2023. arXiv:2311.15127. doi: 10.48550/arXiv.2311.15127Blattmann A., Dockhorn T., Kulal S., Mendelevitch D., Kilian M., Lorenz D., Levi Y., English Z., Voleti V., Letts A., Jampani V., Rombach R. Stable video diffusion: scaling latent video diffusion models to large datasets. arXiv, 2023, arXiv:2311.15127. doi: 10.48550/arXiv.2311.15127Zhang L., Rao A., Agrawala M. Adding conditional control to text-to-image diffusion models // Proc. of the IEEE/CVF International Conference on Computer Vision (ICCV). 2023. P. 3813–3824. doi: 10.1109/ICCV51070.2023.00355Zhang L., Rao A., Agrawala M. Adding conditional control to text-to-image diffusion models. Proc. of the IEEE/CVF International Conference on Computer Vision (ICCV), 2023, pp. 3813–3824. doi: 10.1109/ICCV51070.2023.00355Zhuang S., Li K., Chen X., Wang Y., Liu Z., Qiao Y., Wang Y. Vlogger: make your dream a vlog // Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024. P. 8806–8817. doi: 10.1109/CVPR52733.2024.00841Zhuang S., Li K., Chen X., Wang Y., Liu Z., Qiao Y., Wang Y. Vlogger: make your dream a vlog. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 8806–8817. doi: 10.1109/CVPR52733.2024.00841Xu M., Li H., Su Q., Shang H., Zhang L., Liu C., Wang J., Yao Y., Zhu S. Hallo: hierarchical audio-driven visual synthesis for portrait image animation // arXiv. 2024. arXiv:2406.08801. doi: 10.48550/arXiv.2406.08801Xu M., Li H., Su Q., Shang H., Zhang L., Liu C., Wang J., Yao Y., Zhu S. Hallo: hierarchical audio-driven visual synthesis for portrait image animation. arXiv, 2024, arXiv:2406.08801. doi: 10.48550/arXiv.2406.08801Yang S., Li H., Wu J., Jing M., Li L., Ji R., Liang J., Fan H., Wang J. MegActor-Sigma: unlocking flexible mixed-modal control in portrait animation with diffusion transformer // Proc. of the AAAI Conference on Artificial Intelligence. 2025. V. 39. N 9. P. 9256–9264. doi: 10.1609/aaai.v39i9.33002Yang S., Li H., Wu J., Jing M., Li L., Ji R., Liang J., Fan H., Wang J. MegActor-Sigma: unlocking flexible mixed-modal control in portrait animation with diffusion transformer. Proc. of the AAAI Conference on Artificial Intelligence, 2025, vol.39, no. 9, pp. 9256–9264. doi: 10.1609/aaai.v39i9.33002Lin G., Jiang J., Liang C., Zhong T., Yang J., Zheng Y. CyberHost: taming audio-driven avatar diffusion model with region codebook attention // arXiv. 2024. arXiv:2409.01876. doi: 10.48550/arXiv.2409.01876Lin G., Jiang J., Liang C., Zhong T., Yang J., Zheng Y. CyberHost: taming audio-driven avatar diffusion model with region codebook attention. arXiv, 2024, arXiv:2409.01876. doi: 10.48550/arXiv.2409.01876Chen Z., Cao J., Chen Z., Li Y., Ma C. EchoMimic: lifelike audio-driven portrait animations through editable landmark conditions // Proc. of the AAAI Conference on Artificial Intelligence. 2025. V. 39. N 3. P. 2403–2410. doi: 10.1609/aaai.v39i3.32241Chen Z., Cao J., Chen Z., Li Y., Ma C. EchoMimic: lifelike audio-driven portrait animations through editable landmark conditions. Proc. of the AAAI Conference on Artificial Intelligence, 2025, vol. 39, no. 3, pp. 2403–2410. doi: 10.1609/aaai.v39i3.32241Serengil S., Özpınar A. A benchmark of facial recognition pipelines and co-usability performances of modules // Bilişim Teknolojileri Dergisi. 2024. V. 17. N 2. P. 95–107. doi: 10.17671/gazibtd.1399077Serengil S., Özpınar A. A benchmark of facial recognition pipelines and co-usability performances of modules. Bilişim Teknolojileri Dergisi, 2024, vol. 17, no. 2, pp. 95–107. doi: 10.17671/gazibtd.1399077Bazarevsky V., Kartynnik Y., Vakunov A., Raveendran K., Grundmann M. Blazeface: sub-millisecond seural face detection on mobile GPUs // arXiv. 2019. arXiv:1907.05047. doi: 10.48550/arXiv.1907.05047Bazarevsky V., Kartynnik Y., Vakunov A., Raveendran K., Grundmann M. Blazeface: sub-millisecond seural face detection on mobile GPUs. arXiv, 2019, arXiv:1907.05047. doi: 10.48550/arXiv.1907.05047Zhang F., Bazarevsky V., Vakunov A., Tkachenka A., Sung G., Chang C.L., Grundmann M. MediaPipe hands: on-device real-time hand tracking // arXiv. 2020. arXiv:2006.10214. doi: 10.48550/arXiv.2006.10214Zhang F., Bazarevsky V., Vakunov A., Tkachenka A., Sung G., Chang C.L., Grundmann M. MediaPipe hands: on-device real-time hand tracking. arXiv, 2020, arXiv:2006.10214. doi: 10.48550/arXiv.2006.10214Bazarevsky V., Grishchenko I., Raveendran K., Zhu T., Zhang F., Grundmann M. BlazePose: on-device real-time body pose tracking // arXiv. 2020. arXiv:2006.10204. doi: 10.48550/arXiv.2006.10204Bazarevsky V., Grishchenko I., Raveendran K., Zhu T., Zhang F., Grundmann M. BlazePose: on-device real-time body pose tracking. arXiv, 2020, arXiv:2006.10204. doi: 10.48550/arXiv.2006.10204Xu J., Zou X., Huang K., Chen Y., Liu B., Cheng M., Shi X., Huang J. EasyAnimate: a high-performance long video generation method based on transformer Architecture // arXiv. 2024. arXiv:2405.18991. doi: 10.48550/arXiv.2405.18991Xu J., Zou X., Huang K., Chen Y., Liu B., Cheng M., Shi X., Huang J. EasyAnimate: a high-performance long video generation method based on transformer Architecture. arXiv, 2024, arXiv:2405.18991. doi: 10.48550/arXiv.2405.18991Hu L. Animate anyone: consistent and controllable image-to-video synthesis for character animation // Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2024. P. 8153–8163. doi: 10.1109/CVPR52733.2024.00779Hu L. Animate anyone: consistent and controllable image-to-video synthesis for character animation. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 8153–8163. doi: 10.1109/CVPR52733.2024.00779Ryumina E., Ryumin D., Axyonov A., Ivanko D., Karpov A. Multicorpus emotion recognition method based on cross-modal gated attention fusion // Pattern Recognition Letters. 2025. V. 190. P. 192–200. doi: 10.1016/j.patrec.2025.02.024Ryumina E., Ryumin D., Axyonov A., Ivanko D., Karpov A. Multicorpus emotion recognition method based on cross-modal gated attention fusion. Pattern Recognition Letters, 2025, vol. 190, pp. 192–200. doi:и 10.1016/j.patrec.2025.02.024Peng Y., Sudo Y., Shakeel M., Watanabe S. OWSM-CTC: an open encoder-only speech foundation model for speech recognition, translation, and language identification // Proc. of the 62<sup>nd</sup> Annual Meeting of the Association for Computational Linguistics. 2024. V. 1. P. 10192–10209. doi: 10.18653/v1/2024.acl-long.549Peng Y., Sudo Y., Shakeel M., Watanabe S. OWSM-CTC: an open encoder-only speech foundation model for speech recognition, translation, and language identification. Proc. of the 62<sup>nd</sup> Annual Meeting of the Association for Computational Linguistics, 2024, vol. 1, pp. 10192–10209. doi: 10.18653/v1/2024.acl-long.549Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A., Kaiser L., Polosukhin I. Attention is all you need // Proc. of the Advances in Neural Information Processing Systems 30 (NIPS 2017). 2017. P. 1–11.Vaswani A., Shazeer N., Parmar N., Uszkoreit J., Jones L., Gomez A., Kaiser L., Polosukhin I. Attention is all you need. Proc. of the Advances in Neural Information Processing Systems 30 (NIPS 2017), 2017, pp. 1–11.Kapitanov A., Kvanchiani K., Nagaev A., Petrova E. Slovo: Russian sign language dataset // Lecture Notes in Computer Science. 2023. V. 14253. P. 63–73. doi: 10.1007/978-3-031-44137-0_6Kapitanov A., Kvanchiani K., Nagaev A., Petrova E. Slovo: Russian sign language dataset. Lecture Notes in Computer Science, 2023, vol. 14253, pp. 63–73. doi: 10.1007/978-3-031-44137-0_6Xie L., Wang X., Zhang H., Dong C., Shan Y. VFHQ: a high-quality dataset and benchmark for video face super-resolution // IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2022. P. 657–665. doi: 10.1109/CVPRW56347.2022.00081Xie L., Wang X., Zhang H., Dong C., Shan Y. VFHQ: a high-quality dataset and benchmark for video face super-resolution. IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2022, pp. 657–665. doi: 10.1109/CVPRW56347.2022.00081Kagirov I., Ivanko D., Ryumin D., Axyonov A., Karpov A. TheRuSLan: database of russian sign language // Proc. of the12<sup>th</sup> Conference on Language Resources and Evaluatio (LREC). 2020. P. 6079–6085.Kagirov I., Ivanko D., Ryumin D., Axyonov A., Karpov A. TheRuSLan: database of russian sign language. Proc. of the12<sup>th</sup> Conference on Language Resources and Evaluatio (LREC), 2020. pp. 6079–6085.Кагиров И.А., Рюмин Д.А., Аксенов А.А., Карпов А.А. Мультимедийная база данных жестов русского жестового языка в трехмерном формате // Вопросы языкознания. 2020. № 1. С. 104-123. doi: 10.31857/S0373658X0008302-1Kagirov I., Ryumin D.A., Axyonov A.A., Karpov A.A. Multimedia database of russian sign language items in 3D. Voprosy jazykoznanija, 2020, no. 1., pp. 104–123. (in Russian). doi: 10.31857/S0373658X0008302-1Axyonov A., Ryumin D., Ivanko D., Kashevnik A., Karpov A. Audio-visual speech recognition in-the-wild: multi-angle vehicle cabin corpus and attention-based method // Proc. of the 49<sup>th</sup> IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP). 2024. P. 8195–8199. doi: 10.1109/ICASSP48485.2024.10448048Axyonov A., Ryumin D., Ivanko D., Kashevnik A., Karpov A. Audio-visual speech recognition in-the-wild: multi-angle vehicle cabin corpus and attention-based method. Proc. of the 49<sup>th</sup> IEEE International Conference on Acoustics, Speech, and Signal Processing (ICASSP), 2024, pp. 8195–8199. doi: 10.1109/ICASSP48485.2024.10448048Liu Z. Super Convergence cosine annealing with warm-up learning rate // Proc. of the 2<sup>nd</sup> International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA). 2022. P. 1–7.Liu Z. Super Convergence cosine annealing with warm-up learning rate. Proc. of the 2<sup>nd</sup> International Conference on Artificial Intelligence, Big Data and Algorithms (CAIBDA), 2022, pp. 1–7.Wang P., Shen L., Tao Z., He S., Tao D. Generalization analysis of stochastic weight averaging with general sampling // Proc. of the 41<sup>st</sup> International Conference on Machine Learning (ICML). 2024. P. 51442–51464.Wang P., Shen L., Tao Z., He S., Tao D. Generalization analysis of stochastic weight averaging with general sampling. Proc. of the 41<sup>st</sup> International Conference on Machine Learning (ICML), 2024, pp. 51442–51464.Yang H., Zhang Z., Tang H., Qian J., Yang J. ConsistentAvatar: learning to diffuse fully consistent talking head avatar with temporal guidance // Proc. of the 32<sup>nd</sup> ACM International Conference on Multimedia.2024. P. 3964–3973. doi: 10.1145/3664647.3680619Yang H., Zhang Z., Tang H., Qian J., Yang J. ConsistentAvatar: learning to diffuse fully consistent talking head avatar with temporal guidance. Proc. of the 32<sup>nd</sup> ACM International Conference on Multimedia, 2024, pp. 3964–3973. doi: 10.1145/3664647.3680619Unterthiner T., Van Steenkiste S., Kurach K., Marinier R., Michalski M., Gelly S. Towards accurate generative models of video: a new metric and challenges // arXiv. 2018. arXiv:1812.01717. doi: 10.48550/arXiv.1812.01717Unterthiner T., Van Steenkiste S., Kurach K., Marinier R., Michalski M., Gelly S. Towards accurate generative models of video: a new metric and challenges. arXiv, 2018, arXiv:1812.01717. doi: 10.48550/arXiv.1812.01717Wang Z., Bovik A.C., Sheikh H.R., Simoncelli E.P. Image quality assessment: from error visibility to structural similarity // IEEE Transactions on Image Processing. 2004. V. 13. N 4. P. 600–612. doi: 10.1109/TIP.2003.819861Wang Z., Bovik A.C., Sheikh H.R., Simoncelli E.P. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 2004, vol. 13, no. 4, pp. 600–612. doi: 10.1109/TIP.2003.819861Hore A., Ziou D. Image quality metrics: PSNR vs. SSIM // Proc. of the 20<sup>th</sup> International Conference on Pattern Recognition. 2010. P. 2366–2369. doi: 10.1109/ICPR.2010.579Hore A., Ziou D. Image quality metrics: PSNR vs. SSIM. Proc. of the 20<sup>th</sup> International Conference on Pattern Recognition, 2010, pp. 2366–2369. doi: 10.1109/ICPR.2010.579Deng Y., Yang J., Xu S., Chen D., Jia Y., Tong X. Accurate 3D face reconstruction with weakly-supervised learning: from single image to image set // Proc. of the 32<sup>nd</sup> IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). 2019. P. 285–295. doi: 10.1109/CVPRW.2019.00038Deng Y., Yang J., Xu S., Chen D., Jia Y., Tong X. Accurate 3D face reconstruction with weakly-supervised learning: from single image to image set. Proc. of the 32<sup>nd</sup> IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2019, pp. 285–295. doi: 10.1109/CVPRW.2019.00038Prajwal K.R., Mukhopadhyay R., Namboodiri V.P., Jawahar C.V. A lip sync expert is all you need for speech to lip generation in the wild // Proc. of the 28<sup>th</sup> ACM International Conference on Multimedia. 2020. P. 484–492. doi: 10.1145/3394171.3413532Prajwal K.R., Mukhopadhyay R., Namboodiri V.P., Jawahar C.V. A lip sync expert is all you need for speech to lip generation in the wild. Proc. of the 28<sup>th</sup> ACM International Conference on Multimedia, 2020, pp. 484–492. doi: 10.1145/3394171.3413532

The authors declare that there are no conflicts of interest present.