ntvНаучно-технический вестник информационных технологий, механики и оптикиScientific and Technical Journal of Information Technologies, Mechanics and Optics2226-14942500-0373Университет ИТМО10.17586/2226-1494-2024-24-2-230-240ntv-91Research ArticleКОМПЬЮТЕРНЫЕ СИСТЕМЫ И ИНФОРМАЦИОННЫЕ ТЕХНОЛОГИИCOMPUTER SCIENCEГарантированное обнаружение структурных аномалий в потоковых данных с использованием метода RRCF: выбор параметров обнаружителя и его стабилизация в условиях аддитивных шумовGuarantee structural anomaly detection in streaming data using the RRCF model: selection of detector parameters and its stabilization under additive noise conditionshttps://orcid.org/0000-0001-7212-5230ТимофеевА. В.TimofeevA. V.

Тимофеев Андрей Владимирович — доктор технических наук, научный директор 

 Астана, 010000 

 sc 56689367600 

 Andrey V. Timofeev — D.Sc., Chief Scientific Officer 

 Astana, 010000 

 sc 56689367600 

timofeev.andrey@gmail.comТОО «Эквалайзум»КазахстанLLP “EqualiZoom”Kazakhstan202412122024242230240Copyright © Тимофеев А.В., 20242024Тимофеев А.В.Timofeev A.V.Данная работа распространяется под лицензией Creative Commons Attribution 4.0.This work is licensed under a Creative Commons Attribution 4.0 License.https://ntv.elpub.ru/jour/article/view/91

Введение. Предложены метод стабилизации обнаружения структурных аномалий в условиях аддитивных шумов, а также алгоритм формального выбора параметров решающего правила в обнаружителе структурных аномалий на основе метода Robust Random Cut Forest (RRCF). Метод. В рамках разработанного метода, для стабилизации процесса обнаружения структурных аномалий в условиях воздействия аддитивных шумов, предложено подавать на вход RRCF-обнаружителя поток данных, который предварительно обработан одним из методов цифровой фильтрации. При этом правило принятия решения об обнаружении аномалии строго формализовано и прозрачно интерпретируется. Основные результаты. Формализован выбор параметров стабилизированного методами предварительной фильтрации данных входного потока обнаружителя аномалий на базе RRCF. Параметр обнаружителя, выбранный в рамках предложенный схемы, гарантирует априорно заданную верхнюю границу для вероятности ложной тревоги при принятии решения об обнаружении структурной аномалии. Это свойство строго доказано и оформлено в виде теоремы. Эффективность работы стабилизированного RRCF-обнаружителя аномалий исследована численным методом. Достигнутые результаты подтверждают работоспособность рассмотренного подхода при условии выбора порога обнаружения предложенным способом. Приведен пример практического использования предложенного RRCF-обнаружителя. Обсуждение. Разработанный подход перспективен для обнаружения структурных аномалий в условиях зашумления наблюдений аддитивной помехой, в случае, когда важно гарантировать верхнюю границу для вероятности ложной тревоги. В частности, подход может найти применение при контроле технологических режимов прокачки жидкости в трубопроводных системах или в системах обнаружения предотказных состояний технологического оборудования.

A method for stabilizing structural anomaly detection under additive noise conditions as well as an algorithm for formal selection of the parameters of the solver rule in the structural anomaly detector based on the Robust Random Cut Forest (RRCF) method are proposed. In the framework of the developed approach, in order to stabilize the process of structural anomaly detection under the influence of additive noise, it is proposed to feed to the input of the RRCF-detector a data stream which is pre-processed by one of the digital filtering methods. In this case, the decision rule for anomaly detection is strictly formalized and transparently interpreted. The selection of parameters of the RRCF-based anomaly detector stabilized by pre-filtering methods of the input data stream is formalized. The RRCF-detector parameters choice within the proposed scheme guarantees a predetermined upper bound for the false alarm probability when deciding to detect a structural anomaly. This property is rigorously proved and formalized as a theorem. The performance of the stabilized RRCF-detector is investigated numerically. The achieved results confirm the performance of the proposed approach provided that the detection threshold is selected in the way proposed in this paper. An example of practical application of the proposed method is presented. The developed approach is promising for the detection of structural anomalies in conditions of observation additive noise, in a situation where it is important to guarantee an upper bound for the probability of false alarm. In particular, the approach can find application in monitoring technological regimes of liquid pumping in pipeline systems or in systems for detecting pre-failure states of technological equipment.

Robust Random Cut Forestобнаружение структурных аномалийпотоковая обработка данныхгарантированное обнаружение аномалийRobust Random Cut Foreststructural anomaly detectionstreaming data processingguaranteed anomaly detectionReferencesGomes H.M., Read J., Bifet A. Streaming random patches for evolving data stream classification // Proc. of the IEEE International Conference on Data Mining (ICDM). 2019. P. 240–249. https://doi.org/10.1109/ICDM.2019.00034Gomes H.M., Read J., Bifet A. Streaming random patches for evolving data stream classification // Proc. of the IEEE International Conference on Data Mining (ICDM). 2019. P. 240–249. https://doi.org/10.1109/ICDM.2019.00034Pang Z., Cen J., Yi M. Unsupervised concept drift detection method based on robust random cut forest // International Journal of Machine Learning and Cybernetics. 2023. V. 14. N 12. P. 4207–4222. https://doi.org/10.1007/s13042-023-01890-xPang Z., Cen J., Yi M. Unsupervised concept drift detection method based on robust random cut forest // International Journal of Machine Learning and Cybernetics. 2023. V. 14. N 12. P. 4207–4222. https://doi.org/10.1007/s13042-023-01890-xZheng M., Geng L., Zuo B., Nakata T. A dynamic thresholds based anomaly detection algorithm in energy consumption process of industrial equipment // Proc. of the 2023 7th International Conference on Big Data and Internet of Things. 2023. P. 201–209. https://doi.org/10.1145/3617695.3617706Zheng M., Geng L., Zuo B., Nakata T. A dynamic thresholds based anomaly detection algorithm in energy consumption process of industrial equipment // Proc. of the 2023 7th International Conference on Big Data and Internet of Things. 2023. P. 201–209. https://doi.org/10.1145/3617695.3617706Marathe A. LRZ convolution: An algorithm for automatic anomaly detection in time-series data // Proc. of the 32nd International Conference on Scientific and Statistical Database Management. 2020. P. 1–12. https://doi.org/10.1145/3400903.3400904Marathe A. LRZ convolution: An algorithm for automatic anomaly detection in time-series data // Proc. of the 32nd International Conference on Scientific and Statistical Database Management. 2020. P. 1–12. https://doi.org/10.1145/3400903.3400904Bohlke-Schneider M., Kapoor S., Januschowski T. Resilient neural forecasting systems // Proc. of the Fourth International Workshop on Data Management for End-to-End Machine Learning. 2022. P. 1–5. https://doi.org/10.1145/3399579.3399869Bohlke-Schneider M., Kapoor S., Januschowski T. Resilient neural forecasting systems // Proc. of the Fourth International Workshop on Data Management for End-to-End Machine Learning. 2022. P. 1–5. https://doi.org/10.1145/3399579.3399869Тимофеев А.В. Обнаружение сигналов случайной формы при непараметрической априорной неопределенности относительно распределения наблюдений // Известия вузов. Радиоэлектроника. 1991. № 7. С. 64–68.Timofeev A.V. Detection of randomly shaped signals under nonparametric a priori uncertainty about the distribution of observations. Izvestija vuzov. Radiojelektronika, 1991, no. 7, pp. 64–68. (in Russian)Timofeev A.V., Denisov V.M. Multimodal heterogeneous monitoring of super-extended objects: modern view. recent advances in systems safety and security // Studies in Systems, Decision and Control. 2016. V. 62. P. 97–116. https://doi.org/10.1007/978-3-319-32525-5_6Timofeev A.V., Denisov V.M. Multimodal heterogeneous monitoring of super-extended objects: modern view. recent advances in systems safety and security // Studies in Systems, Decision and Control. 2016. V. 62. P. 97–116. https://doi.org/10.1007/978-3-319-32525-5_6Gomes H., Read J., Bifet A., Barddal J., Gama J. Machine learning for streaming data: state of the art, challenges, and opportunities // ACM SIGKDD Explorations Newsletter. 2019. V. 21. N 2. P. 6–22. https://doi.org/10.1145/3373464.3373470Gomes H., Read J., Bifet A., Barddal J., Gama J. Machine learning for streaming data: state of the art, challenges, and opportunities // ACM SIGKDD Explorations Newsletter. 2019. V. 21. N 2. P. 6–22. https://doi.org/10.1145/3373464.3373470Tatbul N., Lee T., Zdonik S., Alam M., Gottschlich J. Precision and recall for time series // Advances in Neural Information Processing Systems. 2018. V. 31. P. 1924–1934.Tatbul N., Lee T., Zdonik S., Alam M., Gottschlich J. Precision and recall for time series // Advances in Neural Information Processing Systems. 2018. V. 31. P. 1924–1934.Siddiqui M., Fern A., Dietterich T., Wright R., Theriault A., Archer D. Feedback-guided anomaly discovery via online optimization // Proc. of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018. P. 2200–2209. https://doi.org/10.1145/3219819.3220083Siddiqui M., Fern A., Dietterich T., Wright R., Theriault A., Archer D. Feedback-guided anomaly discovery via online optimization // Proc. of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2018. P. 2200–2209. https://doi.org/10.1145/3219819.3220083Hariri S., Kind M. Batch and online anomaly detection for scientific applications in a Kubernetes environment // Proc. of the 9th Workshop on Scientific Cloud Computing. 2018. P. 1–7. https://doi.org/10.1145/3217880.3217883Hariri S., Kind M. Batch and online anomaly detection for scientific applications in a Kubernetes environment // Proc. of the 9th Workshop on Scientific Cloud Computing. 2018. P. 1–7. https://doi.org/10.1145/3217880.3217883Salehi M., Rashidi L. A survey on anomaly detection in evolving data // ACM SIGKDD Explorations Newsletter. 2018. V. 20. N 1. P. 13–23. https://doi.org/10.1145/3229329.3229332Salehi M., Rashidi L. A survey on anomaly detection in evolving data // ACM SIGKDD Explorations Newsletter. 2018. V. 20. N 1. P. 13–23. https://doi.org/10.1145/3229329.3229332Guha S., Mishra N., Roy G., Schrijvers O. Robust random cut forest based anomaly detection on streams // Proceedings of Machine Learning Research. 2016. V. 46. P. 2712–2721.Guha S., Mishra N., Roy G., Schrijvers O. Robust random cut forest based anomaly detection on streams // Proceedings of Machine Learning Research. 2016. V. 46. P. 2712–2721.Breiman L. Bagging predictors // Machine Learning. 1996. V. 24. N 2. P. 123–140. https://doi.org/10.1007/bf00058655Breiman L. Bagging predictors // Machine Learning. 1996. V. 24. N 2. P. 123–140. https://doi.org/10.1007/bf00058655Putina A., Rossi D. Online anomaly detection leveraging streambased clustering and real-time telemetry // IEEE Transactions on Network and Service Management. 2021. V. 18. N 1. P. 839–854. https://doi.org/10.1109/TNSM.2020.3037019Putina A., Rossi D. Online anomaly detection leveraging streambased clustering and real-time telemetry // IEEE Transactions on Network and Service Management. 2021. V. 18. N 1. P. 839–854. https://doi.org/10.1109/TNSM.2020.3037019Vardhan H., Sztipanovits J. Reduced robust random cut forest for out-of-distribution detection in machine learning models // ArXiv. 2022. arXiv:2206.09247. https://doi.org/10.48550/arXiv.2206.09247Vardhan H., Sztipanovits J. Reduced robust random cut forest for out-of-distribution detection in machine learning models // ArXiv. 2022. arXiv:2206.09247. https://doi.org/10.48550/arXiv.2206.09247Arce G.R. Nonlinear Signal Processing: A Statistical Approach. Wiley, 2005. 480 p.Arce G.R. Nonlinear Signal Processing: A Statistical Approach. Wiley, 2005. 480 p.Savitzky A., Golay M.J.E. Smoothing and differentiation of data by simplified least squares procedures // Analytical Chemistry. 1964. V. 36. N 8. P. 1627–1639. https://doi.org/10.1021/ac60214a047Savitzky A., Golay M.J.E. Smoothing and differentiation of data by simplified least squares procedures // Analytical Chemistry. 1964. V. 36. N 8. P. 1627–1639. https://doi.org/10.1021/ac60214a047Тимофеев А.В., Максимов П.Н., Грознов Д.И. Применение оптоволоконной технологии для мониторинга трубопроводных систем отведения шахтных вод в криолитозоне // Гидротехника. 2023. № 3. С. 34–43. https://doi.org/10.55326/22278400_2023_3_34Timofeev A.V., Maksimov P.N., Groznov D.I. Application of fiber optic technology for monitoring the mine water drainage pipeline system in the permafrost zone. The Hydrotechnika, 2023, no. 3, pp . 3 4 – 4 3 . (in Russian) . https://doi.org/10.55326/22278400_2023_3_34

The authors declare that there are no conflicts of interest present.