Feature Distillation

The source code of (Attention-guided Feature Distillation for Semantic Segmentation).

Also, see our previous work (Adaptive Inter-Class Similarity Distillation for Semantic Segmentation).

Requirements

• Python3
• PyTorch (> 0.4.1)
• torchvision
• numpy
• scipy
• tqdm
• matplotlib
• pillow

Datasets and Models

• Datasets: [PascalVoc] [Cityscapes]
• Teacher model: [ResNet101-DeepLabV3+]

Download the datasets and teacher models. Put the teacher model in pretrained/ and set the path to the datasets in mypath.py.

Training

• Without distillation

  python train.py --backbone resnet18 --dataset pascal --nesterov --epochs 120 --batch-size 6

• Distillation

  python train_kd.py --backbone resnet18 --dataset pascal --nesterov --epochs 120 --batch-size 6 --attn_lambda 2

Experimental Results

Comparison of results on the PascalVOC dataset.

Method	mIoU(%)	Params(M)
Teacher: Deeplab-V3 + (ResNet-101)	77.85	59.3
Student: Deeplab-V3 + (ResNet-18)	67.50	16.6
Student + KD	69.13 ± 0.11	16.6
Student + Overhaul	70.67 ± 0.25	16.6
Student + DistKD	69.84 ± 0.11	5.9
Student + CIRKD	71.02 ± 0.11	5.9
Student + LAD	71.42 ± 0.09	5.9
Student + AttnFD (ours)	73.09 ± 0.06	5.9

Comparison of results on the Cityscapes dataset.

Method	mIoU(%)	Accuracy(%)
Teacher: ResNet101	77.66	84.05
Student: ResNet18	64.09	74.8
Student + KD	65.21 (+1.12)	76.32 (+1.74)
Student + Overhaul	70.31 (+6.22)	80.10 (+5.3)
Student + DistKD	71.81 (+7.72)	80.73 (+5.93)
Student + CIRKD	70.49 (+6.40)	79.99 (+5.19)
Student + LAD	71.37 (+7.28)	80.93 (+6.13)
Student + AttnFD (ours)	73.04 (+8.95)	83.01 (+8.21)

Citation

If you use this repository for your research or wish to refer to our distillation method, please use the following BibTeX entry:

@article{mansourian2024attention,
  title={Attention-guided Feature Distillation for Semantic Segmentation},
  author={Mansourian, Amir M and Jalali, Arya and Ahmadi, Rozhan and Kasaei, Shohreh},
  journal={arXiv preprint arXiv:2403.05451},
  year={2024}
}

@article{mansourian2023aicsd,
  title={AICSD: Adaptive Inter-Class Similarity Distillation for Semantic Segmentation},
  author={Mansourian, Amir M and Ahmadi, Rozhan and Kasaei, Shohreh},
  journal={arXiv preprint arXiv:2308.04243},
  year={2023}
}

Acknowledgement

This codebase is heavily borrowed from A Comprehensive Overhaul of Feature Distillation . Thanks for their excellent work.