International Conference on Computer and Knowledge Engineering

Home / 13th International Conference on Computer and Knowledge Engineering

Enhancing Lighter Neural Network Performance with Layer-wise Knowledge Distillation and Selective Pixel Attention

Authors :

Siavash Zaravashan¹ Sajjad Torabi² Hesam Zaravashan³

1- Iran University of Science and Technology 2- Part AI Research Center 3- Iran University of Science and Technology

Keywords :

Knowledge distillation،Image classification،Deep learning،Model compression،Computer vision

Abstract :

Deep neural networks have made a revolution in different areas of Artificial Intelligence such as image classification and object detection. However, most of these models are heavy and time-consuming at the inference stage. If we desire to use real-time applications, it's necessary to lighten them up. However, this could negatively affect the performance of models. One suggested approach for enhancing the capabilities of smaller models involves a technique called knowledge distillation. In this method, a less advanced model, referred to as the student model, learns from and emulates the more advanced teacher model to enhance its performance. This method has been proven as an effective technique to compress the model and increase its accuracy. Previous methods mostly focus on proposing feature transformation and loss functions between the same level’s features to improve the effectiveness. In this paper, we investigate the effect of connections between different layers of teacher and student networks and reveal their great importance. We have employed three different loss functions—attention-driven distillation, hierarchical context loss, and non-local loss—to enhance the classifier's overall performance. Extensive experiments on the CIFAR-100 dataset show that the proposed method has a great performance improvement. For the benefit of the research community, we will make the code for this study available on GitHub.