International Conference on Computer and Knowledge Engineering

Home / 14th International Conference on Computer and Knowledge Engineering

Improve the utility of tensor cores by compacting sparse matrix technique

Authors :

Mohammad.S Abazari¹ Mahsa Zahedi² Abdorreza Savadi³

1- Ferdowsi university of mashhad 2- Ferdowsi university of mashhad 3- Ferdowsi university of mashhad

Keywords :

Tensor Cores،Neural Networks،Convolution Operations،Graphics Processing Unit

Abstract :

Neural networks have demanding computational requirements, particularly in matrix multiplication operations. To address this challenge, we propose a model that combines network pruning and matrix compression techniques. Our approach leverages NVIDIA's tensor cores, which excel at efficient matrix operations. We compress the network weights based on the tensor core structure and perform convolutions using the compressed weight matrix on the tensor cores. Our model incorporates neural network pruning, mixed-precision training, and compression of network weight tensors using the im2col algorithm and CSR format. We also utilize tensor kernels with a block size of 16x16 for multiplication. We evaluate the performance of various models, including pruned, AMP-optimized, combined pruning and AMP techniques, and our proposed model. Our evaluation reveals a significant improvement in performance compared to a simple baseline model. Through an extensive analysis of related works, we establish foundational concepts, present our proposed model, and share the obtained results.