[Deeplab] eval.py/vis.py not working for custom dataset

[AndreiBaraian]

Prerequisites

1. The entire URL of the file you are using

https://github.com/tensorflow/models/tree/master/research/deeplab

2. Describe the bug

I am training deeplab with my custom dataset that has 25 images for training and 11 for testing. It has only one class to detect, so num_classes=2. Image size is 720x2000. Training works fine and I've also set the crop_size to 721x2001. When running eval.py, I get the following error: (0) Invalid argument: assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:] [x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/Cast_1:0) = ] [2]
[[node mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert (defined at /mnt/abandrei/anaconda3/envs/deeplab_tf/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]
[[mean_iou/confusion_matrix/stack_1/_4491]]
(1) Invalid argument: assertion failed: [labels out of bound] [Condition x < y did not hold element-wise:] [x (mean_iou/confusion_matrix/control_dependency:0) = ] [0 0 0...] [y (mean_iou/Cast_1:0) = ] [2]
[[node mean_iou/confusion_matrix/assert_less/Assert/AssertGuard/Assert (defined at /mnt/abandrei/anaconda3/envs/deeplab_tf/lib/python3.7/site-packages/tensorflow_core/python/framework/ops.py:1748) ]]

I've also tried to do the changes in #4203 (comment), but then I get the following error: (0) Invalid argument: Incompatible shapes: [1442721] vs. [1442684]
[[{{node false_negatives_1/Mul}}]]
[[mean_iou/AssignAdd/_4507]]
(1) Invalid argument: Incompatible shapes: [1442721] vs. [1442684]
[[{{node false_negatives_1/Mul}}]]

In this case, I understand that 1442721 is actually 721x2001, but I am not sure about the second shape and what causes it.

3. Steps to reproduce

python3 eval.py --logtostderr --eval_split="val" --model_variant="xception_65" --atrous_rates=6 --atrous_rates=12 --atrous_rates=18 --output_stride=16 --decoder_output_stride=4 --eval_crop_size="721,2001" --eval_logdir=datasets/mydataset/exp/train_on_trainval_set/eval --checkpoint_dir=datasets/mydataset/exp/train_on_trainval_set/train --dataset_dir=datasets/mydataset/tfrecord --eval_batch_size=1 --max_number_of_evaluations=1 --dataset="mydataset"

_mydataset_INFORMATION = DatasetDescriptor(
splits_to_sizes={
'train': 25, # num of samples in images/training
'val': 11, # num of samples in images/validation
},
num_classes=2,
ignore_label=255,
)

4. Expected behavior

The evaluation should be performed.

6. System information

• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 16.04
• TensorFlow installed from (source or binary): binary (pip)
• TensorFlow version (use command below): v1.15.0-rc3-22-g590d6ee 1.15.0
• Python version: 3.7.7
• Bazel version (if compiling from source):
• GCC/Compiler version (if compiling from source):
• CUDA/cuDNN version:10.0/ no cudnn
• GPU model and memory: Tesla P4

[ouzzane]

@AndreiBaraian I had the same problem, did you find a solution

[AndreiBaraian]

@ouzzane yes actually, thank you for reminding, I will also close this issue.

My problem was when providing the ground-truth. Segmentation masks had to be save using np.uint8.
So, make sure that it is something like this: cv2.imwrite(filename, img.astype(np.uint8))

Hope it helps.

[ouzzane]

@AndreiBaraian I didn't count well, it's in the eval.py script?

[AndreiBaraian]

@ouzzane No, separately. It was my script for generating the masks.

[ouzzane]

ok thank you very much for your help @AndreiBaraian

[WisdomPill]

Hello @ouzzane and @AndreiBaraian, I am having the same issue, is there any way to check an image?

[WisdomPill]

Solved it, the script remove_gt_colormap is misleading, it is used for pascal dataset just to convert the images to uint8 with numpy and pil.

I used the following

source_image = Image.open(file_name)
numpy_image = np.array(source_image)

dest_image = Image.fromarray(numpy_image.astype(dtype=np.uint8))
dest_image.save(file_name)

P.S. Remember to check the data of numpy_image