Optimizing knowledge distillation models for language models

The problem of deep neural network compression is discussed using Convolutional Neural Networks (CNNs) as an example. The size of deep neural networks is an obstacle to their practical application under conditions of limited computing resources, energy, and inference latency requirements. One of the developing approaches to compressing deep neural network models is thinning-removing some of the parameters or structural elements of the neural network model. It is shown that thinning is a tradeoff between classification accuracy and computational efficiency. The problem of compressing CNN models was formulated by thinning parameters while maintaining classification quality at the level of the original model, reducing the number of parameters, computational complexity, and inference latency. Standard Top-1 and Top-5 classification accuracy metrics were used to evaluate classification quality. The degree of compression and inference latency were assessed using metrics, such as the number of model parameters, model size, computational complexity, and inference latency. To ensure a fair comparison of the proposed method with others, inference latency measurements were conducted under identical conditions with a fixed input data size. The concept of representation geometry is introduced to control the stability of the internal feature space. The change in the interclass similarity matrix, calculated from the class centroids in the feature space, is used as a metric for preserving the structure of the feature space. The main result of this work is the development of a new compression method for CNNs based on geometrically controlled thinning of the CNN model. In this method, thinning is performed greedily across CNN blocks, and the admissibility of each step is determined not only by the local importance of channels but also by a global constraint on changing the geometry of the representations. The results of the experiment demonstrate that the proposed method consistently maintains classification quality while reducing computational complexity and the number of parameters compared to the baseline method without thinning and the depth-based thinning method of the CNN model. The proposed method for geometrically controlled thinning of a CNN model can be applied to problems requiring real-time decision making as well as on mobile and embedded devices.

1. Dantas P.V., da Silva W.S., Cordeiro L.C., Carvalho C.B. A comprehensive review of model compression techniques in machine learning. Applied Intelligence, 2024, vol. 54, no. 22, pp. 11804-11844. doi: 10.1007/s10489-024-05747-w

2. Bogachev I.V., Bulkanov D.E. Review of modern neural network compression methods for the task of processing measurement data. Bulletin of Pacific National University, 2024, no. 2 (73), pp. 83–92. (in Russian). doi: 10.38161/1996-3440-2024-2-83-92

3. Tatarnikova T.M., Mokretsov N.S. Optimizing knowledge distillation models for language models. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2025, vol. 25, no. 4, pp. 737–743. (in Russian). doi: 10.17586/2226-1494-2025-25-4-737-743

4. Houlsby N., Giurgiu A., Jastrzebski S., Morrone B., de Laroussilhe Q., Gesmundo A., et al. Parameter-efficient transfer learning for NLP. Proc. of the 36th International Conference on Machine Learning, 2019, vol. 97, pp. 2790–2799.

5. Chen J., Hu Y., Zhu J. S-STE: Continuous pruning function for efficient 2:4 sparse pre-training. Proc. of the 38th Conference on Neural Information Processing Systems (NeurIPS), 2024, pp. 33756-33778. doi: 10.52202/079017-1063

6. Frantar E., Alistarh D. SparseGPT: massive language models can be accurately pruned in one-shot. Proc. of the 40th International Conference on Machine Learning, 2023, vol. 202, pp. 10323–10337.

7. Wang Z., Li C., Wang X. Convolutional neural network pruning with structural redundancy reduction. Proc. of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2021, pp. 14908–14917. doi: 10.1109/cvpr46437.2021.01467

8. Chernyshov N.D., Buryak D.YU. Esearch into impact of channel comparison method on efficiency of algorithms for channel-wise pruning of convolutional neural networks. System Analysis in Science and Education, 2025, no. 1, pp. 16–22. (in Russian)

9. Liu D., Zhu Y., Liu Z., Liu Y., Han C., Tian J. A survey of model compression techniques: past, present, and future. Frontiers in Robotics and AI, 2025, vol. 12, pp. 1518965. doi: 10.3389/frobt.2025.1518965

10. He Y., Xiao L. Structured pruning for deep convolutional neural networks: A survey. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, vol. 46, no. 5, pp. 2900–2919. doi: 10.1109/TPAMI.2023.3334614

11. Mokretsov N.S., Tatarnikova T.M. Optimizing the learning process with limited computational resources. International Conference on Soft Computing and Measurement, 2024, vol. 1, pp. 205–208. (in Russian)

12. Lv K., Yang Y., Liu T., Guo Q., Qiu X. Full parameter fine-tuning for large language models with limited resources. Proc. of the 62nd Annual Meeting of the Association for Computational Linguistics, 2024, vol. 1, pp. 8187–8198. doi: 10.18653/v1/2024.acl-long.445

13. Zhang Q., Zhang R., Sun J., Liu Y. How sparse can we prune a deep network: a fundamental limit perspective. Proc. of the 38th International Conference on Neural Information Processing System, 2024, pp. 91337–91372.

14. Cheng H., Zhang M., Shi J.Q. A survey on deep neural network pruning: Taxonomy, comparison, analysis, and recommendations. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, vol. 46, no. 12, pp. 10558–10578. doi: 10.1109/tpami.2024.3447085

15. Zhu K., Hu F., Ding Y., Zhou W., Wang R. A comprehensive review of network pruning based on pruning granularity and pruning time perspectives. Neurocomputing, 2025, vol. 626, pp. 129382. doi: 10.1016/j.neucom.2025.129382

16. Kuzmin V.N., Menisov A.B., Sabirov T.R. A method for optimizing neural networks based on structural distillation using a genetic algorithm. Scientific and Technical Journal of Information Technologies, Mechanics and Optics, 2024, vol. 24, no. 5, pp. 770-778. (in Russian). doi: 10.17586/2226-1494-2024-24-5-770-778

17. Park C., Park M., Moon H., Yoon M.K., Go S., Kim S., et al. DEPrune: Depth-wise separable convolution pruning for maximizing GPU parallelism. Proc. of the 38th International Conference on Neural Information Processing Systems, 2024, pp. 106906–106923.

18. Liao B., Meng Y., Monz C. Parameter-efficient fine-tuning without introducing new latency. Proc. of the 61st Annual Meeting of the Association for Computational Linguistics, 2023, vol. 1, pp. 4242-4260. doi: 10.18653/v1/2023.acl-long.233

19. Lasby M., Golubeva A., Evci U., Nica M., Ioannou Y. Dynamic sparse training with structured sparsity. Proc. of the International Conference on Learning Representations (ICLR), 2024, pp. 32982–33010.

20. Sun X., Shi H. Towards better structured pruning saliency by reorganizing convolution. Proc. of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2024, pp. 2193–2203. doi: 10.1109/WACV57701.2024.00220

21. Wright D., Igel C., Selvan R. BMRS: Bayesian model reduction for structured pruning. Proc. of the 38th International Conference on Neural Information Processing Systems, 2024, pp. 64119–64144.

22. Dong Z., Duan Y., Zhou Y., Duan S., Hu X. Weight-adaptive channel pruning for CNNs based on closeness-centrality modeling. Applied Intelligence, 2024, vol. 54, no. 1, pp. 201–215. doi: 10.1007/s10489-023-05164-5

23. Khan N.A., Rafat A.M.S. Pruning convolution neural networks using filter clustering based on normalized cross-correlation similarity. Journal of Information and Telecommunication, 2025, vol. 9, no. 2, pp. 190–208. doi: 10.1080/24751839.2024.2415008

24. Duan Z., Lu M., Ma J., Huang Y., Ma Z., Zhu F. QARV: QuantizationAware ResNet VAE for lossy image compression. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024, vol. 46, no. 1, pp. 436–450. doi: 10.1109/tpami.2023.3322904