ntvНаучно-технический вестник информационных технологий, механики и оптикиScientific and Technical Journal of Information Technologies, Mechanics and Optics2226-14942500-0373Университет ИТМО10.17586/2226-1494-2025-25-6-1117-1124ntv-544Research ArticleКОМПЬЮТЕРНЫЕ СИСТЕМЫ И ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИCOMPUTER SCIENCEКомбинированная модель качества рекомендательных системCompound quality model for recommender system evaluationhttps://orcid.org/0009-0009-5211-3019ЦыпловА. М.TsyplovA. M.

Алексей Михайлович Цыплов, аспирант

197101; Санкт-Петербург

Aleksei M. Tsyplov, PhD Student

197101; Saint Petersburg

tsyplov80@mail.ruhttps://orcid.org/0000-0003-1588-8164БухановскийА. В.BoukhanovskyA. V.

Александр Валерьевич Бухановский, доктор технических наук,профессор, директор мегафакультета

мегафакультет трансляционных информационных технологий

197101; Санкт-Петербург

sc 6603474810

Alexander V. Boukhanovsky,  D.Sc., Professor, Head of the School

School of Translational Information Technologies

197101; Saint Petersburg

sc 6603474810

avbukhanovskii@itmo.ruУниверситет ИТМОРоссияITMO UniversityRussian Federation20252312202525611171124Copyright © Цыплов А.М., Бухановский А.В., 20252025Цыплов А.М., Бухановский А.В.Tsyplov A.M., Boukhanovsky A.V.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ntv.elpub.ru/jour/article/view/544   Введение

   Введение. Рассмотрены подходы к количественной оценке различных эффектов, таких как позиционный сдвиг (Position Bias), сдвиг в сторону популярных объектов (Popularity Bias) и другие, в рекомендательных системах. Предложена новая модель качества рекомендательных систем, которая приводит выбранные метрики к одной единице измерения и определяет для каждого эффекта его влияние на систему. Полученные оценки позволяют проводить более глубокий сравнительный анализ различных систем, а также исследовать поведение системы на разных сегментах пользователей.

   Метод

   Метод. Для каждой метрики в рамках предложенной модели строится две условные маргинальные плотности распределения: отдельно на релевантных и нерелевантных рекомендациях. На основе сравнения этих плотностей множество возможных значений метрики разделяется на нормальную и критическую. Модель оценивает влияние каждого эффекта на систему на основе частоты попадания значений соответствующей метрики в свою критическую область.

   Основные результаты

   Основные результаты. Для демонстрации работы модели проведен анализ четырех алгоритмов построения рекомендаций на академическом наборе данных MovieLens-100K. В ходе тестирования оценивались Popularity Bias, отсутствие новизны в рекомендациях и склонность систем рекомендовать объекты исключительно на основе демографических данных пользователей. Для каждого эффекта построена оценка его влияния на систему, приведен пример прогнозирования верхней оценки качества системы в случае устранения соответствующего эффекта.

   Обсуждение

   Обсуждение. Показано, что метрики таких эффектов, как Popularity Bias или Position Bias, могут менять распределение абсолютных значений в зависимости от рекомендательной системы. Одним из способов более надежно сравнивать разные рекомендательные системы является предложенная модель качества. Модель подходит для оценивания персональных рекомендаций независимо от сферы применения и алгоритма, который был использован для их построения.

   The study examines approaches to quantifying various effects, such as Position bias, Popularity Bias, and others, in recommender systems. A new quality model of the recommendation algorithms is proposed which reduces the selected metrics to one unit of measurement and determines its impact on the system for each effect. The obtained scores allow for a deeper comparative analysis of various algorithms as well as investigation the behavior of the algorithm in different user segments. For each metric, two conditional marginal distribution densities are built within the framework of the model: separately based on relevant and irrelevant recommendations. Based on the comparison of these densities, the set of possible metric values is divided into normal and critical. The model evaluates the impact of each effect on the system based on the frequency of hitting the values of the corresponding metric in its critical area. To demonstrate how the model works, four recommendation algorithms were analyzed on the MovieLens-100K academic dataset. During the testing, Popularity Bias, the lack of novelty in recommendations, and the tendency of algorithms to recommend objects solely based on user demographic data were evaluated. For each effect, an assessment of its impact on the system is constructed, and an example of predicting an upper estimate of the system quality is given if the corresponding effect is eliminated. The study demonstrated that metrics of effects such as Popularity or Position Bias can change the distribution of absolute values depending on the system. One of the ways to compare different recommendation algorithms more reliably is the proposed quality model. The model is suitable for evaluating personal recommendations, regardless of the scope of application and the algorithm that was used to build them.

рекомендательные системыранжированиеоценка качества рекомендацийPopularity BiasPosition Biasмашинное обучениеrecommendation systemsrankingevaluation of the quality of recommendationspopularity biasposition biasmachine learningReferencesAnderson A., Maystre L., Anderson I., Mehrotra R., Lalmas M. Algorithmic effects on the diversity of consumption on spotify // Proc. of the Web Conference. 2020. P. 2155–2165. doi: 10.1145/3366423.3380281Anderson A., Maystre L., Anderson I., Mehrotra R., Lalmas M. Algorithmic effects on the diversity of consumption on spotify. Proc. of the Web Conference, 2020, pp. 2155–2165. doi: 10.1145/3366423.3380281Avazpour I., Pitakrat T., Grunske L., Grundy J. Dimensions and metrics for evaluating recommendation systems // Recommendation Systems in Software Engineering. 2014. P. 245–273. doi: 10.1007/978-3-642-45135-5_10Avazpour I., Pitakrat T., Grunske L., Grundy J. Dimensions and metrics for evaluating recommendation systems. Recommendation Systems in Software Engineering, 2014, pp. 245–273. doi: 10.1007/978-3-642-45135-5_10Ding H., Kveton B., Ma Y., Park Y., Kini V., Gu Y., et al. Trending now: modeling trend recommendations // Proc. of the 17<sup>th</sup> ACM Conference on Recommender Systems. 2023. P. 294–305. doi: 10.1145/3604915.3608810Ding H., Kveton B., Ma Y., Park Y., Kini V., Gu Y., et al. Trending now: modeling trend recommendations. Proc. of the 17<sup>th</sup> ACM Conference on Recommender Systems, 2023, pp. 294–305. doi: 10.1145/3604915.3608810Cai Y., Guo J., Fan Y., Ai Q., Zhang R., Cheng X. Hard negatives or false negatives: correcting pooling bias in training neural ranking models // Proc. of the 31<sup>st</sup> ACM International Conference on Information and Knowledge Management. 2022. P. 118–127. doi: 10.1145/3511808.3557343Cai Y., Guo J., Fan Y., Ai Q., Zhang R., Cheng X. Hard negatives or false negatives: correcting pooling bias in training neural ranking models. Proc. of the 31<sup>st</sup> ACM International Conference on Information and Knowledge Management, 2022, pp. 118–127. doi: 10.1145/3511808.3557343Abdollahpouri H., Mansoury M., Burke R., Mobasher B. The connection between popularity bias, calibration, and fairness in recommendation // Proc. of the 14<sup>th</sup> ACM Conference on Recommender Systems. 2020. P. 726–731. doi: 10.1145/3383313.3418487Abdollahpouri H., Mansoury M., Burke R., Mobasher B. The connection between popularity bias, calibration, and fairness in recommendation. Proc. of the 14<sup>th</sup> ACM Conference on Recommender Systems, 2020, pp. 726–731. doi: 10.1145/3383313.3418487Beel J., Langer S., Genzmehr M., Gipp B., Breitinger C., Nürnberger A. Research paper recommender system evaluation: a quantitative literature survey // Proc. of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation. 2013. P. 15–22. doi: 10.1145/2532508.2532512Beel J., Langer S., Genzmehr M., Gipp B., Breitinger C., Nürnberger A. Research paper recommender system evaluation: a quantitative literature survey. Proc. of the International Workshop on Reproducibility and Replication in Recommender Systems Evaluation, 2013, pp. 15–22. doi: 10.1145/2532508.2532512Wasilewski J., Hurley N. Incorporating diversity in a learning to rank recommender system // Proc. of the 29<sup>th</sup> International Florida Artificial Intelligence Research Society Conference. 2016. P. 1–6.Wasilewski J., Hurley N. Incorporating diversity in a learning to rank recommender system. Proc. of the 29<sup>th</sup> International Florida Artificial Intelligence Research Society Conference, 2016, pp. 1–6.Ricci F., Rokach L., Shapira B. Recommender Systems Handbook. Springer, 2010. 842 p.Ricci F., Rokach L, Shapira B. Recommender Systems Handbook. Springer, 2010, 842 p.Said A., Bellogin A. Comparative recommender system evaluation: benchmarking recommendation frameworks // Proc. of the 8<sup>th</sup> ACM Conference on Recommender Systems. 2014. P. 129–136. doi: 10.1145/2645710.2645746Said A., Bellogin A. Comparative recommender system evaluation: benchmarking recommendation frameworks. Proc. of the 8<sup>th</sup> ACM Conference on Recommender Systems, 2014, pp. 129–136. doi: 10.1145/2645710.2645746Wilhelm M., Ramanathan A., Bonomo A., Jain S., Chi E.H., Gillenwater J. Practical diversified recommendations on YouTube with determinantal point processes // Proc. of the 27<sup>th</sup> ACM International Conference on Information and Knowledge Management. 2018. P. 2165–2173. doi: 10.1145/3269206.3272018Wilhelm M., Ramanathan A., Bonomo A., Jain S., Chi E.H., Gillenwater J. Practical diversified recommendations on YouTube with determinantal point processes. Proc. of the 27<sup>th</sup> ACM International Conference on Information and Knowledge Management, 2018, pp. 2165–2173 . doi: 10.1145/3269206.3272018Chang Bo, Meng C., Ma H., Chang S., Gu Y., Peng Y., et al. Cluster anchor regularization to alleviate popularity bias in recommender systems // Proc. of the Companion Proceedings of the ACM Web Conference. 2024. P. 151–160. doi: 10.1145/3589335.3648312Chang Bo, Meng C., Ma H., Chang S., Gu Y., Peng Y., et al. Cluster anchor regularization to alleviate popularity bias in recommender systems. Proc. of the Companion Proceedings of the ACM Web Conference, 2024, pp. 151–160. doi: 10.1145/3589335.3648312Bellogin A., Castells P., Cantador I. Precision-oriented evaluation of recommender systems: an algorithmic comparison // Proc. of the 5<sup>th</sup> ACM Conference on Recommender Systems. 2011. P. 333–336. doi: 10.1145/2043932.2043996Bellogin A., Castells P., Cantador I. Precision-oriented evaluation of recommender systems: an algorithmic comparison. Proc. of the 5<sup>th</sup> ACM Conference on Recommender Systems, 2011, pp. 333–336. doi: 10.1145/2043932.2043996Cremonesi P., Koren Y., Turrin R. Performance of recommender algorithms on top-n recommendation tasks // Proc. of the 4<sup>th</sup> ACM Conference on Recommender Systems. 2010. P. 39–46. doi: 10.1145/1864708.1864721Cremonesi P., Koren Y., Turrin R. Performance of recommender algorithms on top-n recommendation tasks. Proc. of the 4<sup>th</sup> ACM Conference on Recommender Systems, 2010, pp. 39–46. doi: 10.1145/1864708.1864721Abdollahpouri H., Burke R., Mobasher B. Managing popularity bias in recommender systems with personalized re-ranking // Proc. of the 32<sup>nd</sup> International Florida Artificial Intelligence Research Society Conference. 2019. P. 1–6.Abdollahpouri H., Burke R., Mobasher B. Managing popularity bias in recommender systems with personalized re-ranking. Proc. of the 32<sup>nd</sup> International Florida Artificial Intelligence Research Society Conference, 2019, pp. 1–6.Yi X., Yang J., Hong L., Cheng D.Z., Heldt L., Kumthekar A., Zhao Z., Wei L., Chi E. Sampling-bias-corrected neural modeling for large corpus item recommendations // Proc. of the 13<sup>th</sup> ACM Conference on Recommender Systems. 2019. P. 269–277. doi: 10.1145/3298689.3346996Yi X., Yang J., Hong L., Cheng D.Z., Heldt L., Kumthekar A., Zhao Z., Wei L., Chi E. Sampling-bias-corrected neural modeling for large corpus item recommendations. Proc. of the 13<sup>th</sup> ACM Conference on Recommender Systems, 2019, pp. 269–277. doi: 10.1145/3298689.3346996Silveira T., Zhang M., Lin X., Liu Y., Ma S. How good your recommender system is? A survey on evaluations in recommendation // International Journal of Machine Learning and Cybernetics. 2019. V. 10. N 5. P. 813–831. doi: 10.1007/s13042-017-0762-9Silveira T., Zhang M., Lin X., Liu Y., Ma S. How good your recommender system is? A survey on evaluations in recommendation. International Journal of Machine Learning and Cybernetics, 2019, vol. 10, no. 5, pp. 813–831. doi: 10.1007/s13042-017-0762-9Akiyama T., Obara K., Tanizaki M. Proposal and evaluation of serendipitous recommendation method using general unexpectedness // CEUR Workshop Proceedings. 2010. V. 676. P. 3–10.Akiyama T., Obara K., Tanizaki M. Proposal and evaluation of serendipitous recommendation method using general unexpectedness. CEUR Workshop Proceedings, 2010, vol. 676, pp. 3–10.Scott L.M., Su-In L. A unified approach to interpreting model predictions // Proc. of the 31<sup>st</sup> Conference on Neural Information Processing Systems. 2017. P. 1–10.Scott L.M., Su-In L. A unified approach to interpreting model predictions. Proc. of the 31<sup>st</sup> Conference on Neural Information Processing Systems, 2017, pp. 1–10.Isinkaye F.O., Folajimi Y.O., Ojokoh B.A. Recommendation systems: principles, methods and evaluation // Egyptian Informatics Journal. 2015. V. 16. N 3. P. 261–273. doi: 10.1016/j.eij.2015.06.005Isinkaye F.O., Folajimi Y.O., Ojokoh B.A. Recommendation systems: principles, methods and evaluation. Egyptian Informatics Journal, 2015, vol. 16, no. 3, pp. 261–273. doi: 10.1016/j.eij.2015.06.005Rhee W., Cho S.-M., Suh B. Countering popularity bias by regularizing score differences // Proc. of the 16<sup>th</sup> ACM Conference on Recommender Systems. 2022. P. 145–155. doi: 10.1145/3523227.3546757Rhee W., Cho S.-M., Suh B. Countering popularity bias by regularizing score differences. Proc. of the 16<sup>th</sup> ACM Conference on Recommender Systems, 2022, pp. 145–155. doi: 10.1145/3523227.3546757Shani G., Gunawardana A. Evaluating recommendation systems // Recommender Systems Handbook. 2010. P. 257–297. doi: 10.1007/978-0-387-85820-3_8Shani G., Gunawardana A. Evaluating recommendation systems. Recommender Systems Handbook, 2010, pp. 257–297. doi: 10.1007/978-0-387-85820-3_8

The authors declare that there are no conflicts of interest present.