Application of neural network and computer vision technologies for image analysis of skin lesion

Opportunity research of using neural networks and computer vision to analyze images of skin lesion and identify features of various pathologies, including oncological neoplasms. A methodology has been developed that makes it possible to evaluate the significance of combinations of color components and spaces in feature extraction using local binary patterns (LBP) and histogram of oriented gradients (HOG) computer vision technologies to extract features of skin changes binary classification of human skin lesions. Optimization of extracted feature makes it possible to more effectively solve the problem of data separability in classification. Research reveals an accessible way to classify skin lesions on a small dataset (less than 1000 images). Research is supposed to be applied to data sequences obtained using a new unique method of multispectral processing of skin lesions. In the course of the work, data from the ISIC-19 and ISIC-20 datasets were used. Samples were formed with a limit of 1000 images for training and validating the models. Additionally, a test sample of 250 images was formed. All images were reduced to 128 × 128 pixels and converted to YCrCb, BGR, Grayscale, HSV color spaces. Features were extracted for each color channel using the HOG and LBP methods. Mathematical models, including neural networks have been used for data classification. The effectiveness of features combinations by color channels and feature extraction methods was evaluated. The preprocessed images were divided into training and validation subsets in a 70/30 ratio. The accuracy, recall, precision and f1-score metrics were used to evaluate the models. The models were evaluated using stratified cross-validation and a test dataset. Optimization of model parameters was carried out based on the loss function represented by the average of cross-validation and evaluation on the validation set. In the process of research, more than 15 000 different optimizations of model parameters were executed. The most stable results on the validation dataset were achieved using ensemble of models, which were trained on a combination of features using local binary patterns (LBP) and histogram of oriented gradients (HOG) technologies. Models which used only local binary patterns technology had the best metrics values, but these models are not recommended to be used in practice without ensemble with stronger models. The results gained can be applied for usage with an ensemble of state-of-the-art convolutional and recurrent neural networks. The proposed approach is universal and applicable both for the analysis of individual images of skin neoplasms and for the analysis of their sequences obtained by the method of multispectral image processing. The technique can be applied to datasets with a limited amount of data. The results obtained will be of interest to specialists in the fields of computer vision and medical images analysis.

1. Zaichenko K.V., Gurevich B.S. Multispectral processing of the biological objects imaging by means of acousto-optic devices. Journal Biomedical Radioelectronics, 2013, no. 9, pp. 70–76. (in Russian)

2. Zaichenko K.V., Gurevich B.S. Application of acousto-optic tunable filters in the devices of skin cancer diagnostics. Proceedings of SPIE, 2020, vol. 11585, pp. 11585OK. https://doi.org/10.1117/12.2581750

3. Zaichenko K.V., Gurevich B.S. Spectral selection using acousto-optic tunable filters for the skin lesions diagnostics. Proceedings of SPIE, 2021, vol. 11922, pp. 119221C. https://doi.org/10.1117/12.2615808

4. Zaichenko K.V., Gurevich B.S. Skin lesions diagnostics by means of multispectral acousto-optic image processing with complexing by x-ray image data. AIP Conference Proceedings, 2020, vol. 2250, pp. 020033. https://doi.org/10.1063/5.0013186

5. Tschandl P., Rosendahl C., Kittler H. The HAM10000 dataset, a large collection of multi-source dermatoscopic images of common pigmented skin lesions. Scientific Data, 2018, vol. 5, pp. 180161. https://doi.org/10.1038/sdata.2018.161

6. Codella N.C.F., Gutman D., Celebi M.E., Helba B., Marchetti M.A., Dusza S.W., Kalloo A., Liopyris K., Mishra N., Kittler H., Halpern A. Skin lesion analysis toward melanoma detection: A challenge at the 2017 International symposium on biomedical imaging (ISBI), hosted by the international skin imaging collaboration (ISIC). Proc. of the 15th IEEE International Symposium on Biomedical Imaging (ISBI), 2018, pp. 168–172. https://doi.org/10.1109/ISBI.2018.8363547

7. Combalia M., Codella N.C.F., Rotemberg V., Helba B., Vilaplana V., Reiter O., Carrera C., Barreiro A., Halpern A.C., Puig S., Malvehyet J. BCN20000: Dermoscopic lesions in the wild. arXiv, 2019, arXiv:1908.02288. https://doi.org/10.48550/arXiv.1908.02288

8. Rotemberg V., Kurtansky N., Betz-Stablein B., Caffery L., Chousakos E., Codella N., Combalia M., Dusza S., Guitera P., Gutman D., Halpern A., Helba B., Kittler H., Kose K., Langer S., Lioprys K., Malvehy J., Musthaq S., Nanda J., Reiter O., Shih G., Stratigos A., Tschandl P., Weber J., Soyer H.P. A patient-centric dataset of images and metadata for identifying melanomas using clinical context. Scientific Data, 2021, vol. 8, no. 1, pp. 34. https://doi.org/10.1038/s41597-021-00815-z

9. Finlayson G., Trezzi E. Shades of gray and colour constancy. Proc. of the IST/SID 12th Color Imaging Conference, 2004, pp. 37–41.

10. Kumar D.M., Babaie M., Zhu S., Kalra S., Tizhoosh H.R. A comparative study of CNN, BoVW and LBP for classification of histopathological images. Proc. of the 2017 IEEE Symposium Series on Computational Intelligence (SSCI), 2017, pp. 1–7. https://doi.org/10.1109/SSCI.2017.8285162

11. Korkmaz S., Akçiçek A., Binol H.B., Korkmaz M. Recognition of the stomach cancer images with probabilistic HOG feature vector histograms by using HOG features. Proc. of the 15th IEEE International Symposium on Intelligent Systems and Informatics (SISY), 2017, pp. 339–342. https://doi.org/10.1109/SISY.2017.8080578

12. Korkmaz S., Binol H. Classification of molecular structure images by using ANN, RF, LBP, HOG, and size reduction methods for early stomach cancer detection. Journal of Molecular Structure, 2018, vol. 1156, pp. 255–263. https://doi.org/10.1016/j.molstruc.2017.11.093

13. Alhakeem Z., Jang S.-I. An LBP-HOG descriptor based on matrix projection for mammogram classification. arXiv, 2021, arXiv.1904.00187. https://doi.org/10.48550/arXiv.1904.00187

14. Agrawal T. Hyperparameter Optimization in Machine Learning. Apress Berkeley, CA, 2021, XIX, 166 p. https://doi.org/10.1007/978-1-4842-6579-6

15. Milantev S., Olyunin V., Bykov I., Milanteva N., Bessmertnyi I. Skin lesion analysis using ensemble of CNN with der moscopic images and metadata. CEUR Workshop Proceedings, 2021, vol. 2893.