CLIPformer

CLIPFormer: Language-Driven Remote Sensing Change Detection with Context-Aware Prompts

Introduction

In this work, we propose a new change detection framework, CLIPFormer, which leverages pretraining knowledge from CLIP and the Swin transformer.

Install

• First, you need to download mmsegmentation and install it on your server.
• Second, Place clipformer.py, swinclip, cswin_text_head.py, and other .py files in the corresponding directory of mmsegmentation..
• Third, train according to the training strategy of mmsegmentation and the training parameters in our paper.

Pretrained Weights of Backbones

CLIP-pretrain Swin-Trnasformer-pretrain

Data Preprocessing

Download the datasets from the official website and split them yourself.

LEVIR-CD LEVIR-CD

LEVIR-CD+ LEVIR-CD+

WHUCD WHUCD

CDD CDD

SYSU-CD SYSU-CD

Training

You can refer to mmsegmentation document (https://mmsegmentation.readthedocs.io/en/latest/index.html).

Results and Logs for CLIPformer

Here we only present the test results of our model. For detailed test results, please refer to our paper.

TABLE I

COMPARISONS OF DETECTION PERFORMANCE ON LEVIR-CD DATASET

Model	Backbone	OA	IoU	F1	Prec	Rec	log
CLIPformer(ViT-B/16)	Swin-T	99.22	85.60	92.24	93.60	90.92	log

TABLE II

COMPARISONS OF DETECTION PERFORMANCE ON LEVIR-CD+ DATASET

Model	Backbone	OA	IoU	F1	Prec	Rec	log
CLIPformer(RN50)	Swin-T	98.87	76.81	86.89	88.51	85.32	log

TABLE III

COMPARISONS OF DETECTION PERFORMANCE ON WHUCD DATASET

Model	Backbone	OA	IoU	F1	Prec	Rec	log
CLIPformer(ViT-B/16)	Swin-T	99.54	89.55	94.49	96.38	92.66	log

TABLE IV

COMPARISONS OF DETECTION PERFORMANCE ON CDD DATASET

Model	Backbone	OA	IoU	F1	Prec	Rec	log
CLIPformer(RN50)	Swin-T	99.33	94.51	97.18	97.03	97.32	log

TABLE V

COMPARISONS OF DETECTION PERFORMANCE ON SYSU-CD DATASET.

Model	Backbone	OA	IoU	F1	Prec	Rec	log
CLIPformer(ViT-B/16)	Swin-T	99.62	71.77	83.57	88.02	79.54	log

Visualization on remote sensing change detection datasets

Here we present the visualization results on the LEVIR-CD dataset. For detailed qualitative analysis and visualization results on other datasets (LEVIR-CD+, WHUCD, CDD, and SYSU-CD), please refer to our paper.

Acknowledgement

Many thanks the following projects's contributions to CLIPFormer.

• MACT-UNet
• mmsegmentation
• DenseCLIP
• CLIP
• ChangeCLIP