Thanks for the great work!
I want to raise a potential issue in the SA-Co eval: it seems like the crowd annotations iscrowd=1 (group masks) are effectively ignored during SA-Co evaluation. That is a discrepancy in how crowd annotations are handled in the custom CGF1Eval class compared to the standard pycocotools.cocoeval.COCOeval evaluator.
In standard COCO evaluation, annotations marked with iscrowd=1 prevent the model from being penalized for predicting individual instances inside a crowd region. A prediction that overlaps with a crowd region is not counted as a False Positive (FP).
However, in the SAM3 codebase, the evaluator completely discards crowd annotations before matching, which causes any prediction overlapping with a crowd region to be penalized as a False Positive.
Here is the inconsistent logic: maskUtils.iou is called with the iscrowd flags in computeIoU(), which calculates the lenient intersection-over-detection-area IoU for crowd columns. However, these computed values are immediately discarded when the columns are sliced out in evaluateImg().
Because of this, models are penalized (FPs increase, precision/cgF1 decreases) for predicting valid instances inside annotated crowd groups. This could be particularly problematic for the Crowded Scenes and Wiki-Food/Drink subsets where 6% of positive image-NP pairs have at least one is_crowd annotation in the GTs.
Thanks for the great work!
I want to raise a potential issue in the SA-Co eval: it seems like the crowd annotations
iscrowd=1(group masks) are effectively ignored during SA-Co evaluation. That is a discrepancy in how crowd annotations are handled in the customCGF1Evalclass compared to the standardpycocotools.cocoeval.COCOevalevaluator.In standard COCO evaluation, annotations marked with
iscrowd=1prevent the model from being penalized for predicting individual instances inside a crowd region. A prediction that overlaps with a crowd region is not counted as a False Positive (FP).However, in the SAM3 codebase, the evaluator completely discards crowd annotations before matching, which causes any prediction overlapping with a crowd region to be penalized as a False Positive.
Here is the inconsistent logic:
maskUtils.iouis called with theiscrowdflags incomputeIoU(), which calculates the lenient intersection-over-detection-area IoU for crowd columns. However, these computed values are immediately discarded when the columns are sliced out inevaluateImg().Because of this, models are penalized (FPs increase, precision/cgF1 decreases) for predicting valid instances inside annotated crowd groups. This could be particularly problematic for the
Crowded ScenesandWiki-Food/Drinksubsets where 6% of positive image-NP pairs have at least oneis_crowdannotation in the GTs.