ntvНаучно-технический вестник информационных технологий, механики и оптикиScientific and Technical Journal of Information Technologies, Mechanics and Optics2226-14942500-0373Университет ИТМО10.17586/2226-1494-2023-23-1-112-120ntv-338Research ArticleИСКУССТВЕННЫЙ ИНТЕЛЛЕКТ И КОГНИТИВНЫЕ ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИARTIFICIAL INTELLIGENCE AND COGNITIVE INFORMATION TECHNOLOGIESПрограммный фреймворк для оптимизации гиперпараметров тематических моделей с аддитивной регуляризациейSoftware framework for hyperparameters optimization of models with additive regularizationhttps://orcid.org/0000-0001-5446-5311ХодорченкоМ. А.KhodorchenkoM. A.

Ходорченко Мария Андреевна - младший научный сотрудник

Санкт-Петербург, 197101

Maria A. Khodorchenko - Junior Researcher

Saint Petersburg, 197101

mariyaxod@yandex.ruhttps://orcid.org/0000-0002-2705-1313БутаковН. А.ButakovN. A.

Бутаков Николай Алексеевич - кандидат технических наук, старший научный сотрудник

Санкт-Петербург, 197101

Nikolay A. Butakov - PhD, Senior Researcher

Saint Petersburg, 197101

alipoov.nb@gmail.comhttps://orcid.org/0000-0002-2740-0173НасоновД. А.NasonovD. A.

Насонов Денис Александрович - кандидат технических наук, старший научный сотрудник

Санкт-Петербург, 197101

Denis A. Nasonov - PhD, Senior Researcher

Saint Petersburg, 197101

denis.nasonov@gmail.comhttps://orcid.org/0000-0001-7114-1052ФируликМ. Ю.FirulikM. Yu.

Фирулик Михаил Юрьевич - директор департамента

Санкт-Петербург, 191028

Mikhail Yu. Firulik - Director of Department

Saint Petersburg, 191028

firulik@gmail.comУниверситет ИТМОРоссияITMO UniversityRussian FederationООО «Оператор Газпром ИД»РоссияLLC “Operator Gazprom ID”Russian Federation202318122024231112120Copyright © Ходорченко М.А., Бутаков Н.А., Насонов Д.А., Фирулик М.Ю., 20242024Ходорченко М.А., Бутаков Н.А., Насонов Д.А., Фирулик М.Ю.Khodorchenko M.A., Butakov N.A., Nasonov D.A., Firulik M.Y.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ntv.elpub.ru/jour/article/view/338

Предмет исследования. Обработка неструктурированных данных, таких как тексты на естественном языке, является одной из актуальных задач при разработке интеллектуальных продуктов. В свою очередь, тематическое моделирование как метод работы с неразмеченными и частично размеченными текстовыми данными активно используется для анализа корпусов документов и создания векторных представлений. В связи с этим особенно важно обучение качественных тематических моделей за короткое время, что возможно с помощью предложенного фреймворка.Метод. Разработанный фреймворк реализует эволюционный подход к оптимизации гиперпараметров моделей с аддитивной регуляризацией и высокими результатами по метрикам качества (когерентность, NPMI). Для уменьшения вычислительного времени представлен режим работы с суррогатными моделями, который обеспечивает ускорение вычислений до 1,8 раз без потери качества.Основные результаты. Эффективность фреймворка продемонстрирована на трех наборах данных с разными статистическими характеристиками. Получены результаты, превосходящие аналогичные решения в среднем на 20 % по когерентности и 5 % по качеству классификации для двух из трех наборов. Создана распределенная версия фреймворка для проведения экспериментальных исследований тематических моделей.Практическая значимость. Полученный фреймворк может быть использован пользователями без специальных знаний в области тематического моделирования, благодаря выстроенному пайплайну работы с данными. Результаты работы могут применяться исследователями для проведения анализа тематических моделей и расширения функционала.

The processing of unstructured data, such as natural language texts, is one of the urgent tasks in the development of intelligent products. In turn, topic modeling as a method of working with unmarked and partially marked text data is a natural choice for analyzing document bodies and creating vector representations. In this regard, it is especially important to train high-quality thematic models in a short time which is possible with the help of the proposed framework. The developed framework implements an evolutionary approach to optimizing hyperparameters of models with additive regularization and high results on quality metrics (coherence, NPMI). To reduce the computational time, a mode of working with surrogate models is presented which provides acceleration of calculations up to 1.8 times without loss of quality. The effectiveness of the framework is demonstrated on three datasets with different statistical characteristics. The results obtained exceed similar solutions by an average of 20 % in coherence and 5 % in classification quality for two of the three datasets. A distributed version of the framework has been developed for conducting experimental studies of topic models. The developed framework can be used by users without special knowledge in the field of topic modeling due to the default data processing pipeline. The results of the work can be used by researchers to analyze topic models and expand functionality.

AutoML фреймворктематическое моделированиенеструктурированные данныеаддитивная регуляризацияэволюционный подходсуррогатные моделиAutoML frameworktopic modelingunstructured dataadditive regularizationevolutionary approachsurrogate modelsИсследование выполнено при финансовой поддержке Российского научного фонда, проект № 20-11-20270.This research is financially supported by The Russian Science Foundation, Agreement No. 20-11-20270.ReferencesKhanthaapha P., Pipanmaekaporn L., Kamonsantiroj S. Topic-based user profile model for POI recommendations // Proc. of the 2nd International Conference on Intelligent Systems, Metaheuristics Swarm Intelligence. 2018. P. 143–147. https://doi.org/10.1145/3206185.3206203Khanthaapha P., Pipanmaekaporn L., Kamonsantiroj S. Topic-based user profile model for POI recommendations. Proc. of the 2nd International Conference on Intelligent Systems, Metaheuristics Swarm Intelligence, 2018, pp. 143–147. https://doi.org/10.1145/3206185.3206203Peña F.J., O’Reilly-Morgan D., Tragos E.Z., Hurley N., Duriakova E., Smyth B., Lawlor A. Combining rating and review data by initializing latent factor models with topic models for Top-N recommendation // Proc. of the 14th ACM Conference on Recommender Systems. 2020. P. 438–443. https://doi.org/10.1145/3383313.3412207Peña F.J., O’Reilly-Morgan D., Tragos E.Z., Hurley N., Duriakova E., Smyth B., Lawlor A. Combining rating and review data by initializing latent factor models with topic models for top-n recommendation. Proc. of the 14th ACM Conference on Recommender Systems, 2020, pp. 438–443. https://doi.org/10.1145/3383313.3412207Sokhin T., Butakov N. Semi-automatic sentiment analysis based on topic modeling // Procedia Computer Science. 2018. V. 136. P. 284–292. https://doi.org/10.1016/j.procs.2018.08.286Sokhin T., Butakov N. Semi-automatic sentiment analysis based on topic modeling. Procedia Computer Science, 2018, vol. 136, pp. 284–292. https://doi.org/10.1016/j.procs.2018.08.286Nevezhin E., Butakov N., Khodorchenko M., Petrov M., Nasonov D. Topic-driven ensemble for online advertising generation // Proc. of the 28th International Conference on Computational Linguistics. 2020. P. 2273–2283. https://doi.org/10.18653/v1/2020.coling-main.206Nevezhin E., Butakov N., Khodorchenko M., Petrov M., Nasonov D. Topic-driven ensemble for online advertising generation. Proc. of the 28th International Conference on Computational Linguistics, 2020, pp. 2273–2283. https://doi.org/10.18653/v1/2020.coling-main.206Zamiralov A., Khodorchenko M., Nasonov D. Detection of housing and utility problems in districts through social media texts // Procedia Computer Science. 2020. V. 178. P. 213–223. https://doi.org/10.1016/j.procs.2020.11.023Zamiralov A., Khodorchenko M., Nasonov D. Detection of housing and utility problems in districts through social media texts. Procedia Computer Science, 2020, vol. 178, pp. 213–223. https://doi.org/10.1016/j.procs.2020.11.023Shi T., Kang K., Choo J., Reddy C.K. Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations // Proc. of the World Wide Web Conference (WWW 2018). 2018. P. 1105–1114. https://doi.org/10.1145/3178876.3186009Shi T., Kang K., Choo J., Reddy C.K. Short-text topic modeling via non-negative matrix factorization enriched with local word-context correlations. Proc. of the World Wide Web Conference (WWW 2018), 2018, pp. 1105–1114. https://doi.org/10.1145/3178876.3186009Hofmann T. Probabilistic latent semantic indexing // Proc. of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘99). 1999. P. 50–57. https://doi.org/10.1145/312624.312649Hofmann T. Probabilistic latent semantic indexing. Proc. of the 22nd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval (SIGIR ‘99), 1999, pp. 50–57. https://doi.org/10.1145/312624.312649Blei D.M., Ng A.Y., Jordan M.I. Latent Dirichlet allocation // Journal of Machine Learning Research. 2003. V. 3. P. 993–1022.Blei D.M., Ng A.Y., Jordan M.I. Latent Dirichlet allocation. Journal of Machine Learning Research, 2003, vol. 3, pp. 993–1022.Vorontsov K., Potapenko A., Plavin A. Additive regularization of topic models for topic selection and sparse factorization // Lecture Notes in Computer Science. 2015. V. 9047. P. 193–202. https://doi.org/10.1007/978-3-319-17091-6_14Vorontsov K., Potapenko A., Plavin A. Additive regularization of topic models for topic selection and sparse factorization. Lecture Notes in Computer Science (including subseries Lecture Notes in Artificial Intelligence and Lecture Notes in Bioinformatics), 2015, vol. 9047, pp. 193–202. https://doi.org/10.1007/978-3-319-17091-6_14Card D., Tan C., Smith N.A. Neural models for documents with metadata // Proc. of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2018. P. 2031–2040. https://doi.org/10.18653/v1/p18-1189Card D., Tan C., Smith N.A. Neural models for documents with metadata. Proc. of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 2018, pp. 2031–2040. https://doi.org/10.18653/v1/p18-1189Cao Z., Li S., Liu Y., Li W., Ji H. A novel neural topic model and its supervised extension // Proceedings of the AAAI Conference on Artificial Intelligence. 2015. V. 29. N 1. P. 2210–2216. https://doi.org/10.1609/aaai.v29i1.9499Cao Z., Li S., Liu Y., Li W., Ji H. A novel neural topic model and its supervised extension. Proceedings of the AAAI Conference on Artificial Intelligence, 2015, vol. 29, no. 1, pp. 2210–2216. https:// doi.org/10.1609/aaai.v29i1.9499Bianchi F., Terragni S., Hovy D. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence // Proc. of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers). 2021. P. 759–766. https://doi.org/10.18653/v1/2021.acl-short.96Bianchi F., Terragni S., Hovy D. Pre-training is a hot topic: Contextualized document embeddings improve topic coherence. Proc. of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 2: Short Papers), 2021, pp. 759–766. https://doi.org/10.18653/v1/2021.acl-short.96Ye J., Jing X., Li J. Sentiment Analysis Using Modified LDA // Lecture Notes in Electrical Engineering. 2018. V. 473. P. 205–212. https://doi.org/10.1007/978-981-10-7521-6_25Ye J., Jing X., Li J. Sentiment Analysis Using Modified LDA. Lecture Notes in Electrical Engineering, 2018, vol. 473, pp. 205–212. https:// doi.org/10.1007/978-981-10-7521-6_25Bodrunova S., Koltsov S., Koltsova O., Nikolenko S., Shimorina A. Interval semi-supervised LDA: Classifying needles in a haystack // Lecture Notes in Computer Science. 2013. V. 8265. P. 265–274. https://doi.org/10.1007/978-3-642-45114-0_21Bodrunova S., Koltsov S., Koltsova O., Nikolenko S., Shimorina A. Interval semi-supervised LDA: Classifying needles in a haystack. Lecture Notes in Computer Science, 2013, vol. 8265, pp. 265–274. https://doi.org/10.1007/978-3-642-45114-0_21Řehůřek R., Sojka P. Software framework for topic modelling with large corpora // Proc. of the LREC 2010 Workshop on New Challenges for NLP. 2010. P. 45–50.Řehůřek R., Sojka P. Software framework for topic modelling with large corpora. Proc. of the LREC 2010 Workshop on New Challenges for NLP, 2010, pp. 45–50.Terragni S., Fersini E., Galuzzi B.G., Tropeano P., Candelieri A. OCTIS: Comparing and optimizing topic models is simple! // Proc. of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations. 2021. P. 263–270. https://doi.org/10.18653/v1/2021.eacl-demos.31Terragni S., Fersini E., Galuzzi B.G., Tropeano P., Candelieri A. OCTIS: Comparing and optimizing topic models is simple! Proc. of the 16th Conference of the European Chapter of the Association for Computational Linguistics: System Demonstrations, 2021, pp. 263–270. https://doi.org/10.18653/v1/2021.eacl-demos.31Khodorchenko M., Butakov N. Developing an approach for lifestyle identification based on explicit and implicit features from social media // Procedia Computer Science. 2018. V. 136. P. 236–245. https://doi.org/10.1016/j.procs.2018.08.262Khodorchenko M., Butakov N. Developing an approach for lifestyle identification based on explicit and implicit features from social media. Procedia Computer Science, 2018, vol. 136, pp. 236–245. https://doi.org/10.1016/j.procs.2018.08.262Vorontsov K., Frei O., Apishev M., Romov P., Dudarenko M. BigARTM: Open source library for regularized multimodal topic modeling of large collections // Communications in Computer and Information Science. 2015. V. 542. P. 370–381. https://doi.org/10.1007/978-3-319-26123-2_36Vorontsov K., Frei O., Apishev M., Romov P., Dudarenko M. BigARTM: Open source library for regularized multimodal topic modeling of large collections. Communications in Computer and Information Science, 2015, vol. 542, pp. 370–381. https://doi.org/10.1007/978-3-319-26123-2_36Khodorchenko M., Teryoshkin S., Sokhin T., Butakov N. Optimization of learning strategies for ARTM-based topic models // Lecture Notes in Computer Science. 2020. V. 12344. P. 284–296. https://doi.org/10.1007/978-3-030-61705-9_24Khodorchenko M., Teryoshkin S., Sokhin T., Butakov N. Optimization of learning strategies for ARTM-based topic models. Lecture Notes in Computer Science, 2020, vol. 12344, pp. 284–296. https://doi.org/10.1007/978-3-030-61705-9_24Khodorchenko M., Butakov N., Sokhin T., Teryoshkin S. Surrogatebased optimization of learning strategies for additively regularized topic models // Logic Journal of the IGPL. 2022. https://doi.org/10.1093/jigpal/jzac019Khodorchenko M., Butakov N., Sokhin T., Teryoshkin S. Surrogatebased optimization of learning strategies for additively regularized topic models. Logic Journal of the IGPL, 2022. https://doi.org/10.1093/jigpal/jzac019Röder M., Both A., Hinneburg A. Exploring the space of topic coherence measures // Proc. of the Eighth ACM International Conference on Web Search and Data Mining (WSDM’15). 2015. P. 399–408. https://doi.org/10.1145/2684822.2685324Röder M., Both A., Hinneburg A. Exploring the space of topic coherence measures. Proc. of the Eighth ACM International Conference on Web Search and Data Mining (WSDM’15), 2015, pp. 399–408. https://doi.org/10.1145/2684822.2685324Newman D., Noh Y., Talley E., Karimi S., Baldwin T. Evaluating topic models for digital libraries // Proc. of the 10th Annual Joint Conference on Digital Libraries (JCDL’10). 2010. P. 215–224. https:// doi.org/10.1145/1816123.1816156Newman D., Noh Y., Talley E., Karimi S., Baldwin T. Evaluating topic models for digital libraries. Proc. of the 10th Annual Joint Conference on Digital Libraries (JCDL’10), 2010, pp. 215–224. https://doi.org/10.1145/1816123.1816156

The authors declare that there are no conflicts of interest present.