Please wait ...
0% Complete
Home
/
13th International Conference on Computer and Knowledge Engineering
Enhancing Lighter Neural Network Performance with Layer-wise Knowledge Distillation and Selective Pixel Attention
Authors :
Siavash Zaravashan
1
Sajjad Torabi
2
Hesam Zaravashan
3
1- Iran University of Science and Technology
2- Part AI Research Center
3- Iran University of Science and Technology
Keywords :
Knowledge distillation،Image classification،Deep learning،Model compression،Computer vision
Abstract :
Deep neural networks have made a revolution in different areas of Artificial Intelligence such as image classification and object detection. However, most of these models are heavy and time-consuming at the inference stage. If we desire to use real-time applications, it's necessary to lighten them up. However, this could negatively affect the performance of models. One suggested approach for enhancing the capabilities of smaller models involves a technique called knowledge distillation. In this method, a less advanced model, referred to as the student model, learns from and emulates the more advanced teacher model to enhance its performance. This method has been proven as an effective technique to compress the model and increase its accuracy. Previous methods mostly focus on proposing feature transformation and loss functions between the same level’s features to improve the effectiveness. In this paper, we investigate the effect of connections between different layers of teacher and student networks and reveal their great importance. We have employed three different loss functions—attention-driven distillation, hierarchical context loss, and non-local loss—to enhance the classifier's overall performance. Extensive experiments on the CIFAR-100 dataset show that the proposed method has a great performance improvement. For the benefit of the research community, we will make the code for this study available on GitHub.
Papers List
List of archived papers
To Transfer or Not To Transfer (TNT): Action Recognition in Still Image Using Transfer Learning
Ali Soltani Nezhad - Hojat Asgarian Dehkordi - Seyed Sajad Ashrafi - Shahriar Baradaran Shokouhi
Automatic Generation of XACML Code using Model-Driven Approach
Athareh Fatemian - Bahman Zamani - Marzieh Masoumi - Mehran Kamranpour - Behrouz Tork Ladani - Shekoufeh Kolahdouz Rahimi
A Graph-based Feature Selection using Class-Feature Association Map (CFAM)
Motahare Akhavan - Seyed Mohammad Hossein Hasheminejad
A Comprehensive Dataset of Real-scene Images for Text Detection and Recognition in Persian
Iman Souzanchi - Ramin Rahimi - Mohammad Ali Majidi Anvari - Atefeh Baniasadi - Ashkan Sadeghi - Mohammad Reza Mohammadi
PeQa: a Massive Persian Quenstion-Answering and Chatbot Dataset
Fatemeh Zahra Arshia - Mohammad Ali Keyvanrad - Saeedeh Sadat Sadidpour - Sayyid Mohammad Reza Mohammadi
The Effect of Network Environment on Traffic Classification
Abolghasem Rezaei Khesal - Mehdi Teimouri
Parallel Local Feature Selection For High-dimensional Data
Zhaleh Manbari - Chiman Salavati - Fardin AkhlaghianTab - Barzan Saeedpoor - Himan Delbina - Mahmud Abdulla Mohammad
Automated software design using Machine Learning With Natural Language Processing
Fahimeh Khedmatkon - Seyed Mohammad Hossein Hasheminejad - Jaleh Shoshtarian Malak
A Review on Machine Learning Methods for Workload Prediction in Cloud Computing
Mohammad Yekta - Hadi Shahriar Shahhoseini
Evaluation of Efficient Electrocardiomatrix-based Identification Using Deep Learning Methods
Amirhossein Safari - Narges Mokhtari - Mohsen Hooshmand - Sadegh Sadeghi - Peyman Pahlevani
more
Samin Hamayesh - Version 46.0.2