Real-time segmentation

[Kowasaki]

Hey @gustavz, as mentioned is the previous threads I am looking into efficient segmentation networks that can be run on the TX2. Would you mind looping me in to the conversation regarding the mask implementation? Me and rest of the team at my university are all very interested in this subject and would like to figure out the best approach for doing real-time segmentation.

[gustavz]

@Kowasaki I am currently working my way through the following projects / papers to come up with a ssd modification that allows segmentation in parallel to bounding box detection.
Please feel free to add code proposals or other sources if you find some.
I will keep you up to date as i proceed.

• https://github.com/soo89/Rainbow-SSD
• https://github.com/kmlee-snu/run
• https://github.com/hszhao/PSPNet
• https://github.com/rykov8/ssd_keras
• https://github.com/pierluigiferrari/ssd_keras
• https://github.com/mrgloom/awesome-semantic-segmentation

[Kowasaki]

Thanks for the links--I will be looking through them as well!

I'm also wondering if you are open to plain old segmentation models? I've found a model that is meant to achieve 7fps on the TX1: https://github.com/Eromera/erfnet_pytorch

I'll need to test it out after reading up on PyTorch but assuming the GPU RAM usage isn't overwhelming I'm considering running both the detector and the segmentation model at once.

[gustavz]

yeah you can do this ofcourse! its a valid workaround, but i want to achieve a network that can do both simultaniously :)

[Kowasaki]

Gotcha--in this case you might want to take at look at the differences between the Box predictors used for the SSD and the Mask-RCNN networks here: https://github.com/tensorflow/models/blob/master/research/object_detection/core/box_predictor.py.

I suspect you can add segmentation for the SSD boxes to use some sort of encoder-decoder network like you mentioned before. The FCN on the Mask-RCNN stuff is way too heavy for real-time. I'll have to try this later as well but unfortunately time constraints dictate I stick with what works for now!

[gustavz]

@Kowasaki any news about this topic from your side?

[Kowasaki]

@gustavz Unfortunately no. The SSDlite-Deeplab model described in the mobilenet v2 paper is the best bet I think, but I haven't had time to look over how to implement that whole thing.

Have you found anything that works well with SSD? The SSD-Lite part described in the paper may be worth a shot.