Mask identity jumping between similar objects

[tywei08]

Hi all,

I’m encountering an issue when using SAM2 for multi-object video segmentation/tracking, and I’m wondering whether others have seen similar behavior or have suggestions to mitigate it.

Setup

Task: Generate masklets for fruit flies in a video

Method:

1. Provide an initial mask prompt on the first frame
2. Use SAM2 to propagate masks across the video

Objects: Multiple fruit flies with very similar appearance

Observed issue

The predicted mask for a given object occasionally jumps to a different fly, even when:

• The two flies are not spatially close
• There is no obvious occlusion or interaction
• Once the jump happens, the identity is effectively lost and continues tracking the wrong fly

I’ve attached a short video demonstrating the issue. In the video, the initial mask is correct, but at certain frames the mask abruptly switches to another fly elsewhere in the scene.

github_idtracker_drosophila_6.mp4

Since the flies have very similar appearance, I suspect this may be an identity ambiguity issue in the model, but I’m wondering whether others have observed similar behavior and if there are any known workarounds or best practices to reduce it. Otherwise, I’m curious if this is a known limitation of the current model design.

[heyoeyo]

I've definitely seen it jump to different objects, but usually they're close by. It's strange to see it jump to something so far away.

The tracking uses the last 6 frames (by default) along with the prompted frame to do tracking, so it seems like if the initial prompt was in the lower right, and a similar looking fly ends up in that position, the prompt 'memory' might cause the tracking to switch back to the lower right.
If that were the case, then one solution would be to 're-prompt' the tracking repeatedly over time, so that it's more heavily biased towards choosing locations closer to where the fly has been recently.

Another solution might be to use something like SAMURAI, which basically just keeps a separate tracker (kalman filter) running and so can detect when there's disagreement between SAM and the kalman filter (and then prevent a jump like this).