Whence MaskRCNN's segm IoU metrics = 0?

Question

When training a MaskRCNN on my multi-class instance segmentation custom data set, given an input formatted as:

image   -)  shape: torch.Size([3, 850, 600]),   dtype: torch.float32, min: tensor(0.0431),               max: tensor(0.9137)
boxes   -)  shape: torch.Size([4, 4]),          dtype: torch.float32, min: tensor(47.),                  max: tensor(807.)
masks   -)  shape: torch.Size([850, 600, 600]), dtype: torch.uint8,   min: tensor(0, dtype=torch.uint8), max: tensor(1, dtype=torch.uint8)
areas   -)  shape: torch.Size([4]),             dtype: torch.float32, min: tensor(1479.),                max: tensor(8014.)
labels  -)  shape: torch.Size([4]),             dtype: torch.int64,   min: tensor(1),                    max: tensor(1)
iscrowd -)  shape: torch.Size([4]),             dtype: torch.int64,   min: tensor(0),                    max: tensor(0)

I consistently obtain all segmentation IoU metrics as shown below:

DONE (t=0.03s).
IoU metric: bbox
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.004
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.010
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.004
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.001
IoU metric: segm
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.75      | area=   all | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Precision  (AP) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=  1 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets= 10 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=   all | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= small | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area=medium | maxDets=100 ] = 0.000
 Average Recall     (AR) @[ IoU=0.50:0.95 | area= large | maxDets=100 ] = 0.000

How can I think, debug and fix this?

Answer 1

As your input image size is (850, 600) (H, W) and considering that for this given image you have 4 objects, not 850 with (600, 600) masks. your masks tensor should have dimension (number of objects, 850, 600), thus your input should be:

image   -)  shape: torch.Size([3, 850, 600]),   dtype: torch.float32, min: tensor(0.0431),               max: tensor(0.9137)
boxes   -)  shape: torch.Size([4, 4]),          dtype: torch.float32, min: tensor(47.),                  max: tensor(807.)
masks   -)  shape: torch.Size([4, 850, 600]), dtype: torch.uint8,   min: tensor(0, dtype=torch.uint8), max: tensor(1, dtype=torch.uint8)
areas   -)  shape: torch.Size([4]),             dtype: torch.float32, min: tensor(1479.),                max: tensor(8014.)
labels  -)  shape: torch.Size([4]),             dtype: torch.int64,   min: tensor(1),                    max: tensor(1)
iscrowd -)  shape: torch.Size([4]),             dtype: torch.int64,   min: tensor(0),                    max: tensor(0)

How to fix it Because you are trying to solve an instance segmentation problem, ensure that each of your (850, 600) masks are stacked so as to yield a tensor in the (number of masks, 850, 600) shape.