This is the source code for our paper: Channel Pruning Guided by Spatial and Channel Attention for DNNs in Intelligent Edge Computing. A brief introduction of this work is as follows:
Deep Neural Networks (DNNs) have achieved remarkable success in many computer vision tasks recently, but the huge number of parameters and the high computation overhead hinder their deployments on resource-constrained edge devices. It is worth noting that channel pruning is an effective approach for compressing DNN models. A critical challenge is to determine which channels are to be removed, so that the model accuracy will not be negatively affected. In this paper, we first propose Spatial and Channel Attention (SCA), a new attention module combining both spatial and channel attention that respectively focuses on "where" and "what" are the most informative parts. Guided by the scale values generated by SCA for measuring channel importance, we further propose a new channel pruning approach called Channel Pruning guided by Spatial and Channel Attention (CPSCA). Experimental results indicate that SCA achieves the best inference accuracy, while incurring negligibly extra resource consumption, compared to other state-of-the-art attention modules. Our evaluation on two benchmark datasets shows that, with the guidance of SCA, our CPSCA approach achieves higher inference accuracy than other state-of-the-art pruning methods under the same pruning ratios.
深度神经网络（DNN）近来在众多计算机视觉任务中取得显著成功，但其海量参数和高计算开销阻碍了其在资源受限边缘设备上的部署。值得注意的是，通道剪枝是一种有效的DNN模型压缩方法。其核心挑战在于确定哪些通道可以被移除，从而确保模型精度不受负面影响。本文首先提出空间与通道注意力（SCA）模块，这是一种融合空间注意力（聚焦"何处"是最具信息量的区域）与通道注意力（聚焦"什么"是最重要特征）的新型注意力机制。基于SCA生成的用于衡量通道重要性的尺度值指导，我们进一步提出了一种名为"基于空间与通道注意力的通道剪枝"（CPSCA）的新方法。实验结果表明：与其他最先进的注意力模块相比，SCA在仅增加微不足道的额外资源消耗的同时，实现了最佳的推理精度。在两个基准数据集上的评估表明：在SCA的指导下，我们的CPSCA方法在相同剪枝率条件下，比其他最先进的剪枝方法实现了更高的推理精度。
This paper has been accepted and has been published by Applied Soft Computing (ASOC), and the preprint version can be downloaded from here. You can also download the formal version from here.
We only provide our SCA and CPSCA here. You can find the implementation of other attention models mentioned in our paper from PytorchInsight. Due to some reason, we didn't provide the scaler_for_prune.txt file required by prune.py in the released code. If you want to know how to generate it, please contact the 1st author with [email protected].
PyTorch
If you use these models in your research, please cite:
@article{LIU2021107636,
title = {Channel pruning guided by spatial and channel attention for DNNs in intelligent edge computing},
journal = {Applied Soft Computing},
volume = {110},
pages = {107636},
year = {2021},
issn = {1568-4946},
doi = {https://doi.org/10.1016/j.asoc.2021.107636},
url = {https://www.sciencedirect.com/science/article/pii/S1568494621005573},
author = {Mengran Liu and Weiwei Fang and Xiaodong Ma and Wenyuan Xu and Naixue Xiong and Yi Ding},
}
Mengran Liu ([email protected])
Please note that the open source code in this repository was mainly completed by the graduate student author during his master's degree study. Since the author did not continue to engage in scientific research work after graduation, it is difficult to continue to maintain and update these codes. We sincerely apologize that these codes are for reference only.