Can we support multi-bands image as input in Detectron2?

[sqwev]

I'm a student major in agricultural remote sensing, so I think multispectral remote sensing images play an important role in agricultrue scenes. However, I found Derectron2 has a strict limit in input image, the input images should be uint8 and 3or 4 bands, which is incompatible with many satellite images. Do you have a plan to make Detectron2 support more input types in the future?

[aradfarahani]

Hi there!

Detectron2, as it currently stands, primarily supports images with 3 or 4 channels (typically RGB or RGBA) and expects them to be in the uint8 format. This limitation can indeed be restrictive for applications involving multispectral or other types of remote sensing images that often have more than 4 bands and may not be in the uint8 format.

Current Workarounds and Discussions

1. Custom Data Loaders: One way to handle multi-band images is to create custom data loaders that can preprocess and feed the images into the model. This approach requires modifying the data loading pipeline to handle the specific format and number of channels of your images.

2. Community Discussions: There have been discussions in the Detectron2 community about supporting different image formats and more channels. For example, users have explored ways to train with float32 images and 2-channel images by modifying the data loading and preprocessing steps.

3. Future Plans: While there is no official statement from the Detectron2 team about expanding support to more input types, the community's interest and contributions could potentially lead to more flexible input handling in future updates.

Example of Custom Data Loader

Here is a simplified example of how you might modify the data loader to handle multi-band images:

import torch
from detectron2.data import DatasetMapper, build_detection_train_loader
from detectron2.data import detection_utils as utils

class CustomDatasetMapper(DatasetMapper):
    def __call__(self, dataset_dict):
        dataset_dict = copy.deepcopy(dataset_dict)
        image = utils.read_image(dataset_dict["file_name"], format="BGR")
        # Custom processing for multi-band images
        # For example, if your image has 6 bands:
        image = image[:, :, :6]  # Assuming the image has at least 6 bands
        dataset_dict["image"] = torch.as_tensor(image.transpose(2, 0, [1](https://github.com/facebookresearch/detectron2/discussions/2512)).astype("float32"))
        return dataset_dict

# Use the custom mapper in the data loader
data_loader = build_detection_train_loader(cfg, mapper=CustomDatasetMapper(cfg, is_train=True))

This example assumes you have a custom dataset where each image has more than the standard 3 or 4 channels. You would need to adjust the preprocessing steps according to your specific requirements.

Conclusion

While Detectron2 does have limitations regarding input image formats and channels, there are ways to work around these constraints using custom data loaders. The community's ongoing discussions and contributions may also lead to more flexible support in the future.