How to use the Segment Anything Model (SAM) to save the annotated Image Segments (as COCO format)?

Question

The demo of the Segment Anything Model (SAM) is very impressive (as shown below) and opens up a bunch of possibilities. I have a dataset that I was about to annotate using normal bounding boxes, but now I want to use segmentation instead, and annotate it with the use of the much more precise SAM.

I'm not asking about the training aspect at all here. I just want a simple tool or method to save my masks and assign a class to them and preferably export it as COCO (or another annotation format) files. Is there currently an easy way to simply export the masks and assign a class to the, per image. Without involving, setting up whole environment or a lot of coding. Preferably an open source and free to use tool that I can install, or use in browser. Or a Docker container to spin-up.

Yes, I could develop it myself, and spend hours on it, but as it's a very small side-project and I have no time for that, so I am searching for an easy and quick solution to use this for annotating images and preparing a dataset for training. The only solutions I found so far are paid 'as a service' models that charge hundreds of dollars a month.

Answer 1

Could you also add to your question whether you want a tool which enables segmentation creation upfront quickly (advanced labelling tool)? OR something that you can run over an already created set of bounding boxes (just a python script).

CVAT can do this, but it's not very easy to setup. I'll try to come back here with some more detailed instructions soon, but for the time-being, here are some links which are better than nothing.

First you need to set up auto-annotation (which is not made obvious in there post about SAM)

https://opencv.github.io/cvat/docs/administration/advanced/installation_automatic_annotation/

Then you need to deploy SAM specifically once you have nuclio working (container which runs the serverless functions):

./serverless/deploy_cpu.sh serverless/pytorch/facebookresearch/sam

https://www.cvat.ai/post/facebook-segment-anything-model-in-cvat