A PyTorch library for developing energy efficient multiplication-less models.

TorchShiftAdd Overview

Welcome to TorchShiftAdd, your go-to open-source library for crafting energy-efficient multiplication-less models and applications!

TorchShiftAdd embodies a pioneering initiative to simplify and expand the realm of multiplication-less networks within the machine learning community. Key features include:

• Ready-to-use implementation of a wide range of ShiftAdd-based multiplication-less CNNs or Transformers.
• CUDA kernels and TVM compilation support for seamless GPU deployment.
• Profiling tools to furnish FLOPs, energy, and latency breakdown data for in-depth analysis and optimization.
• Hardware accelerator simulators to estimate energy savings and latency improvements on ASICs or FPGAs.
• Flexible support for developing both algorithmic and hardware accelerator designs tailored for multiplication-less networks.

List of Implemented Papers

• ShiftAdd-based Convolutional Neural Networks
  • [NeurIPS'20] ShiftAddNet: A Hardware-Inspired Deep Network
  • [CVPR'20 Oral] AdderNet: Do We Really Need Multiplications in Deep Learning?
  • [CVPR'21 Workshop] DeepShift: Towards Multiplication-Less Neural Networks
  • [ICLR'23] Bit-Pruning: A Sparse Multiplication-Less Dot-Product
• ShiftAdd-based Transformers
  • [NeurIPS'23] ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer
  • [NeurIPS'22] EcoFormer: Energy-Saving Attention with Linear Complexity
• Linear Attention in Transformers
  • [CVPR'23] Castling-ViT: Compressing Self-Attention via Switching Towards Linear-Angular Attention During Vision Transformer Inference
  • [HPCA'23] ViTALiTy: Unifying Low-rank and Sparse Approximation for Vision Transformer Acceleration with a Linear Taylor Attention
• Hardware Accelerators for ShiftAdd-based Multiplication-less Networks
  • [ICCAD'22] NASA: Neural Architecture Search and Acceleration for Hardware Inspired Hybrid Networks
  • [IEEE TCAS-I] NASA+: Neural Architecture Search and Acceleration for Multiplication-Reduced Hybrid Networks

Installation

Coming soon.

Qucik Start

Coming soon.

Contributing

TorchShiftAdd is released under Apache-2.0 License. Everyone is welcome to contribute to the development of TorchShiftAdd. Please refer to contributing guidelines for more details.

Acknowledgement

Coming soon.