Precision annotation for bounding boxes, using SAM or SAM2

[Youho99]

Actions before raising this issue

• I searched the existing issues and did not find anything similar.
• I read/searched the docs

Is your feature request related to a problem? Please describe.

Today, when we want to annotate objects precisely with bounding boxes, it is necessary to zoom in on the object to be almost pixel-perfect, especially for small objects.

This is extremely time-consuming.

The idea of ​​this feature is to be able to annotate quickly and precisely with bounding boxes.

I've already proposed this feature on #8243 and #8231

Describe the solution you'd like

The idea is to make an imprecise bounding box around the object to be annotated. The bounding box is sent to the SAM or SAM2 model, which segments the main object from the bounding box it receives. Finally, the precise bounding box is recreated by taking the extremum coordinates at the top, left, bottom, right.

This would allow very quick and precise annotating, without having to zoom in on the image (very useful for precise annotation of small objects for example).

Describe alternatives you've considered

In my free time, I made a python script using this logic with SAM to make precision annotation, taking as input an annotation json (COCO format I think) and which output a json in the same format, with the precise bounding boxes recalculated.

I could make it available to you if necessary.

Additional context

Many other ideas can arise from this idea, such as:

• Automatically correct the annotations of an imprecisely annotated dataset (whose bounding boxes are larger than the object). This dataset can come from the internet for example, as you can find plenty of them on Roboflow.

• Quickly and roughly annotate a few objects (without correcting the precision), train a model which will be poorly performing in its metrics, but which will be capable of producing imprecise bounding boxes (larger than the object), and finally automatically correct the precision of these automatically generated bounding boxes. This would enable accurate self-labeling models to be obtained in a short time.

[bsekachev]

Hello,

1. Annotate masks/polygons using SAM/SAM2 (e.g. including bounding box as input, what was recently supported in Segment Anything: supported bounding box input #8270)
2. Run shapes converter to convert annotated shapes to rectangles (on app.cvat.ai) or postprocess final polygons with your own scripts

[Youho99]

@bsekachev Thanks for your message !

Great, I didn't know about this possibility.

I was thinking of a direct solution instead, but that could also do the trick initially.

[Youho99]

So, reconsidering the feature that I am proposing, with the "shape converters" part, it would be interesting to have the "rectangle to SAM masks" option. Because currently, the "rectangle to mask" option creates a square mask, like the bbox...

This could make it possible to process and make accurate datasets with imprecise annotations in one go.

[Youho99]

One last point: Is the behavior you cited reproducible with the Dockerized version?

[bsekachev]

I was thinking of a direct solution instead, but that could also do the trick initially.

Sure, that just was workaround proposed.