Compression Resources

!! Note

For a more comprehensive list of DNN compression research papers this Google Sheet I created. Comments can be made directly in the Google Sheet, and are very welcome.

Element-wise pruning

• Song Han, Huizi Mao, and William J Dally. Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding. 2015

Group Lasso regularization

• Noah Simon , Jerome Friedman , Trevor Hastie & Robert Tibshirani. A Sparse-Group Lasso. Journal of Computational and Graphical Statistics, Volume 22, 2013 - Issue 2
• Wei Wen, Chunpeng Wu, Yandan Wang, Yiran Chen, and Hai Li. Learning Structured Sparsity in Deep Neural Networks. Neural Information Processing Systems, 2016
• Alireza Aghasi, Afshin Abdi, Nam Nguyen, and Justin Romberg. Net-Trim: Convex Pruning of Deep Neural Networks with Performance Guarantee. 2016.
• Simone Scardapane, Danilo Comminiello, Amir Hussain, and Aurelio Uncini. Group sparse regularization for deep neural networks. Neurocomputing, 241:81–89, 2017

RNNs

• Sharan Narang, Erich Elsen, Gregory Diamos, Shubho Sengupta. Exploring Sparsity in Recurrent Neural Networks. ICLR, 2017
• Artem M. Grachev, Dmitry I. Ignatov, Andrey V. Savchenko. Neural Networks Compression for Language Modeling. 2017
• Zhe Li, Shuo Wang, Caiwen Ding, Qinru Qiu, Yanzhi Wang, Yun Liang. Efficient Recurrent Neural Networks using Structured Matrices in FPGAs. ICLR Workshop, 2018
• Wei Wen, Yuxiong He, Samyam Rajbhandari, Minjia Zhang, Wenhan Wang, Fang Liu, Bin Hu, Yiran Chen, Hai Li. Learning Intrinsic Sparse Structures within Long Short-Term Memory. ICLR, 2018.

Pruning using Reinforcement Learning

• Anubhav Ashok, Nicholas Rhinehart, Fares Beainy, Kris M. Kitani. N2N Learning: Network to Network Compression via Policy Gradient Reinforcement Learning. 2017
• Franco Manessi, Alessandro Rozza, Simone Bianco, Paolo Napoletano, Raimondo Schettini. Automated Pruning for Deep Neural Network Compression. 2017
• Yihui He, Song Han. ADC: Automated Deep Compression and Acceleration with Reinforcement Learning. 2018
• Barret Zoph and Quoc V. Le. Neural Architecture Search with Reinforcement Learning. In ICLR, pages 976–981, nov 2017.

Uncategorized

• Zhuang Liu, Jianguo Li, Zhiqiang Shen, Gao Huang, Shoumeng Yan, and Changshui Zhang. Learning Efficient Convolutional Networks through Network Slimming. ICCV 2017, 2017
• Jose M Alvarez and Mathieu Salzmann. Learning the Number of Neurons in Deep Networks. Neural Information Processing Systems, 2016.
• Jose M Alvarez and Mathieu Salzmann. Compression-aware Training of Deep Networks. In Neural Information Processing Systems, pages 1–10, 2017.
• Hao Zhou, Jose M Alvarez, and Fatih Porikli. Less Is More: Towards Compact CNNs. Computer Vision – ECCV 2016, pages 662–677, 2016.
• Jaehong Yoon and Sung Ju Hwang. Combined Group and Exclusive Sparsity for Deep Neural Networks. Proceedings of the 34th International Conference on Machine Learning, 70:3958– 3966, 2017.