Description
While I was deep diving into the pycocotools code I have found that it is not possible to distinguish a missed detection and a correct detection of the first ground truth (index 0).
This is because the dtMatches (dtm) array is instantiated as np.zeros.
According to the results I get from faster_coco_eval the same behavior affects also this library.
This problem should not affect the metrics in practice with realistic dataset sizes, moreover fixing it will break the compatibility with pycocotools results. It is just confusing if you debug the results at the single sample or run the metrics on toy datasets.
I have also found that this can be easily solved with 1-index annotations, this is probably a requirement of COCO API I missed, probably inherited from the original Matlab implementation.
To Reproduce
Create a single image dataset with two annotations with id 0 and 1, the maximum TP will be 1 (precision 0.5).
Expected behavior
I think the easiest way to keep compatibility with pycocotools is to raise a warning if the annotations start from index 0, this can happen with custom datasets.
Additional context
The bug is also reported here cocodataset/cocoapi#664
Description
While I was deep diving into the
pycocotoolscode I have found that it is not possible to distinguish a missed detection and a correct detection of the first ground truth (index 0).This is because the
dtMatches(dtm) array is instantiated asnp.zeros.According to the results I get from
faster_coco_evalthe same behavior affects also this library.This problem should not affect the metrics in practice with realistic dataset sizes, moreover fixing it will break the compatibility with
pycocotoolsresults. It is just confusing if you debug the results at the single sample or run the metrics on toy datasets.I have also found that this can be easily solved with 1-index annotations, this is probably a requirement of COCO API I missed, probably inherited from the original Matlab implementation.
To Reproduce
Create a single image dataset with two annotations with
id0 and 1, the maximum TP will be 1 (precision 0.5).Expected behavior
I think the easiest way to keep compatibility with
pycocotoolsis to raise a warning if the annotations start from index 0, this can happen with custom datasets.Additional context
The bug is also reported here cocodataset/cocoapi#664