A multivariate binary decision tree classifier based on shallow neural network
https://doi.org/10.17586/2226-1494-2022-22-4-725-733
Abstract
In this paper, a novel decision tree classifier based on shallow neural networks with piecewise and nonlinear transformation activation functions are presented. A shallow neural network is recursively employed into linear and non-linear multivariate binary decision tree methods which generates splitting nodes and classifier nodes. Firstly, a linear multivariate binary decision tree with a shallow neural network is proposed which employs a rectified linear unit function. Secondly, there is presented a new activation function with non-linear property which has good generalization ability in learning process of neural networks. The presented method shows high generalization ability for linear and non-linear multivariate binary decision tree models which are called a Neural Network Decision Tree (NNDT). The proposed models with high generalization ability ensure the classification accuracy and performance. A novel split criterion of generating the nodes which focuses more on majority objects of classes on the current node is presented and employed in the new NNDT models. Furthermore, a shallow neural network based NNDT models are converted into a hyperplane based linear and non-linear multivariate decision trees which has high speed in the processing classification decisions. Numerical experiments on publicly available datasets have showed that the presented NNDT methods outperform the existing decision tree algorithms and other classifier methods.
Keywords
hierarchical classifier,
neural networks,
binary tree,
multivariate decision tree,
activation function
Supporting agencies: This research is supported by the Department of Algorithms and Programming technologies, National University of Uzbekistan, Uzbekistan.
About the Authors
A. R. Marakhimov
Termez State University
Uzbekistan
Uzbekistan
Avazjon R. Marakhimov — D. Sc., Professor, Rector
Termez, 190011
J. K. Kudaybergenov
Tashkent University of Information Technologies Nukus branch named after Muhammad Al-Khwarizmi
Uzbekistan
Uzbekistan
Jabbarbergen K. Kudaybergenov — PhD, Lecturer
Nukus, 230113
K. K. Khudaybergenov
National University of Uzbekistan
Uzbekistan
Uzbekistan
Kabul K. Khudaybergenov — PhD, Lecturer
Tashkent, 100174
U. R. Ohundadaev
National University of Uzbekistan
Uzbekistan
Uzbekistan
Ulugbek R. Ohundadaev — Basic Doctoral Student
Tashkent, 100174
References
1. Morala P., Cifuentes J.A., Lillo R.E., Ucar I. Towards a mathematical framework to inform neural network modelling via polynomial regression. Neural Networks, 2021, vol. 142, pp. 57–72. https://doi.org/10.1016/j.neunet.2021.04.036
2. Cao W., Mirjalili V., Raschka S. Rank consistent ordinal regression for neural networks with application to age estimation. Pattern Recognition Letters, 2020, vol. 140, pp. 325–331. https://doi.org/10.1016/j.patrec.2020.11.008
3. Messner E., Fediuk M., Swatek P., Scheidl S., Smolle-Jüttner F.M., Olschewski H., Pernkopf F. Multi-channel lung sound classification with convolutional recurrent neural networks. Computers in Biology and Medicine, 2020, vol. 122, pp. 103831. https://doi.org/10.1016/j.compbiomed.2020.103831
4. Youling L. A calibration method of computer vision system based on dual attention mechanism. Image and Vision Computing, 2020, vol. 103, pp. 104039. https://doi.org/10.1016/j.imavis.2020.104039
5. Palmerston J.B., Zhou Y., Chan H.M. Comparing biological and artificial vision systems: Network measures of functional connectivity. Neuroscience Letters, 2020, vol. 739, pp. 135407. https://doi.org/10.1016/j.neulet.2020.135407
6. Basha S.H.Sh., Dubey Sh.R., Pulabaigari V., Mukherjee S. Impact of fully connected layers on performance of convolutional neural networks for image classification. Neurocomputing, 2020, vol. 378, pp. 112–119. https://doi.org/10.1016/j.neucom.2019.10.008
7. Shuang K., Tan Y., Cai Zh., Sun Y. Natural language modeling with syntactic structure dependency. Information Sciences, 2020, vol. 523, pp. 220–233. https://doi.org/10.1016/j.ins.2020.03.022
8. Xu M. WITHDRAWN: Image processing system based on FPGA and convolutional neural network. Microprocessors and Microsystems, 2020, pp. 103379. https://doi.org/10.1016/j.micpro.2020.103379
9. Krizhevsky A., Sutskever I., Hinton G.E. Imagenet classification with deep convolutional neural networks. Proc. of the 26th Annual Conference on Neural Information Processing Systems (NIPS), 2012, pp. 1097–1105.
10. Goodfellow I., Bengio Y., Courville A. Deep Learning. MIT Press, 2016, 775 p.
11. Zhou Z.H., Feng J. Deep forest: Towards an alternative to deep neural networks. Proc. of the 26th International Joint Conference on Artificial Intelligence (IJCAI), 2017, pp. 3553–3559. https://doi.org/10.24963/ijcai.2017/497
12. Frosst N., Hinton G. Distilling a neural network into a soft decision tree. CEUR Workshop Proceedings, 2018, vol. 2070.
13. Wan A., Dunlap L., Ho D., Yin J., Lee S., Jin H., Petryk S., Bargal S.A., Gonzalez J.E. NBDT: Neural-backed decision trees. arXiv, 2020, arXiv:2004.00221. https://doi.org/10.48550/arXiv.2004.00221
14. Pinto A., Pereira S., Rasteiro D.M., Silva C. Hierarchical brain tumour segmentation using extremely randomized trees. Pattern Recognition, 2018, vol. 82, pp. 105–117. https://doi.org/10.1016/j.patcog.2018.05.006
15. Vanli N.D., Sayin M.O., Mohaghegh N.M., Ozkan H., Kozat S.S. Nonlinear regression via incremental decision trees. Pattern Recognition, 2019, vol. 86, pp. 1–13. https://doi.org/10.1016/j.patcog.2018.08.014
16. Blanco-Justici A., Domingo-Ferrer J., Martínez S., Sánchez D. Machine learning explainability via microaggregation and shallow decision trees. Knowledge-Based Systems, 2020, vol. 194, pp. 105532. https://doi.org/10.1016/j.knosys.2020.105532
17. Kotsiantis S.B. Decision trees: a recent overview. Artificial Intelligence Review, 2013, vol. 39, no. 4, pp. 261–283. https://doi. org/10.1007/s10462-011-9272-4
18. Quinlan J.R. Induction of decision trees. Machine Learning, 1986, vol. 1, no. 1, pp. 81–106. https://doi.org/10.1023/A:1022643204877
19. Breiman L., Friedman J.H., Stone C.J., Olshen R.A. Classification and Regression Trees. Chapman & Hall/CRC, 1984.
20. Quinlan J.R. C4.5: Programs for Machine Learning. Morgan Kaufmann Publishers Inc., 1993.
21. Chandra B., Paul Varghese P. Moving towards efficient decision tree construction. Information Sciences, 2009, vol. 179, no. 8, pp. 1059–1069. https://doi.org/10.1016/j.ins.2008.12.006
22. Wang F., Wang Q., Nie F., Yu W., Wang R. Efficient tree classifiers for large scale datasets. Neurocomputing, 2018, vol. 284, pp. 70–79. https://doi.org/10.1016/j.neucom.2017.12.061
23. Yildiz C., Alpaydin E. Omnivariate decision trees. IEEE Transactions on Neural Networks, 2001, vol. 12, no. 6, pp. 1539–1546. https://doi.org/10.1109/72.963795
24. Altinçay H. Decision trees using model ensemble-based nodes. Pattern Recognition, 2007, vol. 40, no. 12, pp. 3540–3551. https:// doi.org/10.1016/j.patcog.2007.03.023
25. Kumar M.A., Gopal M. A hybrid SVM based decision tree. Pattern Recognition, 2010, vol. 43, no. 12, pp. 3977–3987. https://doi.org/10.1016/j.patcog.2010.06.010
26. Nie F., Zhu W., Li X. Decision Tree SVM: An extension of linear SVM for non-linear classification. Neurocomputing, 2020, vol. 401, pp. 153–159. https://doi.org/10.1016/j.neucom.2019.10.051
Review
For citations:
Marakhimov A.R., Kudaybergenov J.K., Khudaybergenov K.K., Ohundadaev U.R. A multivariate binary decision tree classifier based on shallow neural network. Scientific and Technical Journal of Information Technologies, Mechanics and Optics. 2022;22(4):725-733. https://doi.org/10.17586/2226-1494-2022-22-4-725-733
Views:
10
Email this article 