[WIP] Polygon tracking feature with Cutie #8261
Conversation
|
Important Review skippedAuto incremental reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughThis update enhances the CVAT framework by introducing a polygon tracking functionality in the annotation tools, allowing users to track objects using polygons instead of just rectangles. Additionally, a new evaluation configuration and logging setups are introduced for model evaluation and performance monitoring, improving the overall usability and efficiency of the system for video object segmentation tasks. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant ToolsControl
participant ModelHandler
User->>ToolsControl: Initiate Polygon Tracking
ToolsControl->>ModelHandler: Send Image and Polygon Data
ModelHandler->>ModelHandler: Process Image for Tracking
ModelHandler->>ToolsControl: Return Updated Shapes
ToolsControl->>User: Display Updated Tracking Information
Assessment against linked issues
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 6
Outside diff range, codebase verification and nitpick comments (1)
serverless/pytorch/redefine/cutie/nuclio/main.py (1)
1-3: Add a brief description of the file.Consider adding a brief description at the beginning of the file to provide context about its purpose.
# SPDX-License-Identifier: MIT # # This file contains the main logic for the Cutie model handler.
Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files ignored due to path filters (1)
serverless/pytorch/redefine/cutie/nuclio/sample/sample.jpgis excluded by!**/*.jpg
Files selected for processing (9)
- cvat-ui/src/components/annotation-page/standard-workspace/controls-side-bar/tools-control.tsx (3 hunks)
- serverless/pytorch/redefine/cutie/nuclio/config/eval_config.yaml (1 hunks)
- serverless/pytorch/redefine/cutie/nuclio/config/hydra/job_logging/custom-no-rank.yaml (1 hunks)
- serverless/pytorch/redefine/cutie/nuclio/config/hydra/job_logging/custom.yaml (1 hunks)
- serverless/pytorch/redefine/cutie/nuclio/config/model/base.yaml (1 hunks)
- serverless/pytorch/redefine/cutie/nuclio/function-gpu.yaml (1 hunks)
- serverless/pytorch/redefine/cutie/nuclio/function.yaml (1 hunks)
- serverless/pytorch/redefine/cutie/nuclio/main.py (1 hunks)
- serverless/pytorch/redefine/cutie/nuclio/model_handler.py (1 hunks)
Additional context used
yamllint
serverless/pytorch/redefine/cutie/nuclio/config/hydra/job_logging/custom-no-rank.yaml
[error] 22-22: no new line character at the end of file
(new-line-at-end-of-file)
serverless/pytorch/redefine/cutie/nuclio/config/hydra/job_logging/custom.yaml
[error] 22-22: no new line character at the end of file
(new-line-at-end-of-file)
serverless/pytorch/redefine/cutie/nuclio/function.yaml
[error] 72-72: no new line character at the end of file
(new-line-at-end-of-file)
Additional comments not posted (29)
serverless/pytorch/redefine/cutie/nuclio/config/model/base.yaml (6)
1-2: LGTM! Pixel statistics are correctly defined.The pixel mean, standard deviation, and dimension are standard values for image normalization.
Also applies to: 4-4
10-12: LGTM! Encoder settings are correctly defined.The pixel and mask encoder types and dimensions are appropriate for the model configuration.
Also applies to: 14-16
18-19: LGTM! Positional encoding settings are correctly defined.The scale and temperature values are appropriate for positional encoding.
21-41: LGTM! Transformer settings are correctly defined.The embedding dimensions, feed-forward dimensions, number of heads, blocks, queries, and attention settings are appropriate for the transformer configuration.
43-46: LGTM! Summarizer settings are correctly defined.The embedding dimensions, number of summaries, and positional encoding are appropriate for the summarizer configuration.
48-54: LGTM! Loss and decoder settings are correctly defined.The sensory and query loss settings with weights and the upsampling dimensions for the decoder are appropriate for the configuration.
Also applies to: 56-58
serverless/pytorch/redefine/cutie/nuclio/function.yaml (6)
1-13: LGTM! Metadata is correctly defined.The name, namespace, annotations, and help message are appropriate for the function configuration.
15-23: LGTM! Specifications are correctly defined.The description, runtime, handler, event timeout, and environment variables are appropriate for the function configuration.
24-55: LGTM! Build directives are correctly defined.The image, base image, and various run commands for setting up the environment and installing dependencies are appropriate for the function configuration.
56-62: LGTM! Triggers are correctly defined.The HTTP trigger with max workers and attributes is appropriate for the function configuration.
63-65: LGTM! Resource limits are correctly defined.The GPU limits are appropriate for the function configuration.
67-72: LGTM! Platform settings are correctly defined.The restart policy and mount mode are appropriate for the function configuration.
Tools
yamllint
[error] 72-72: no new line character at the end of file
(new-line-at-end-of-file)
serverless/pytorch/redefine/cutie/nuclio/function-gpu.yaml (6)
1-13: LGTM! Metadata is correctly defined.The name, namespace, annotations, and help message are appropriate for the function configuration.
15-23: LGTM! Specifications are correctly defined.The description, runtime, handler, event timeout, and environment variables are appropriate for the function configuration.
24-58: LGTM! Build directives are correctly defined.The image, base image, and various run commands for setting up the environment and installing dependencies are appropriate for the function configuration.
59-63: LGTM! Triggers are correctly defined.The HTTP trigger with max workers and attributes is appropriate for the function configuration.
64-68: LGTM! Resource limits are correctly defined.The GPU limits are appropriate for the function configuration.
70-75: LGTM! Platform settings are correctly defined.The restart policy and mount mode are appropriate for the function configuration.
serverless/pytorch/redefine/cutie/nuclio/config/eval_config.yaml (5)
1-4: LGTM!The
defaultssection correctly references the base configurations and overrides.
6-10: LGTM!The
hydrasection correctly specifies the run directory and output subdirectory.
11-23: LGTM!The main configuration parameters are correctly specified and make sense in the context of the evaluation configuration.
25-34: LGTM!The dataset-specific parameters are correctly specified and make sense in the context of the evaluation configuration.
54-147: LGTM!The dataset configurations are correctly specified and make sense in the context of the evaluation configuration.
serverless/pytorch/redefine/cutie/nuclio/model_handler.py (3)
15-29: LGTM!The
convert_polygon_to_maskfunction correctly converts a polygon to a mask.
31-73: LGTM!The
convert_mask_to_polygonfunction correctly converts a mask to a polygon.
74-93: LGTM!The
prepare_numpy_imagefunction correctly converts a numpy image to a torch tensor with the correct shape and dtype.cvat-ui/src/components/annotation-page/standard-workspace/controls-side-bar/tools-control.tsx (3)
507-507: LGTM!The change correctly updates the shape type from
ShapeType.RECTANGLEtoShapeType.POLYGON.
595-595: LGTM!The change correctly updates the shape type filter from
shapeType === 'rectangle'toshapeType === 'polygon'.
1049-1074: LGTM!The changes correctly update the user interface to include separate buttons for "Track Rectangle" and "Track Polygon."
| # python logging configuration for tasks | ||
| version: 1 | ||
| formatters: | ||
| simple: | ||
| format: '[%(asctime)s][%(levelname)s] - %(message)s' | ||
| datefmt: '%Y-%m-%d %H:%M:%S' | ||
| handlers: | ||
| console: | ||
| class: logging.StreamHandler | ||
| formatter: simple | ||
| stream: ext://sys.stdout | ||
| file: | ||
| class: logging.FileHandler | ||
| formatter: simple | ||
| # absolute file path | ||
| filename: ${hydra.runtime.output_dir}/${now:%Y-%m-%d_%H-%M-%S}-eval.log | ||
| mode: w | ||
| root: | ||
| level: INFO | ||
| handlers: [console, file] | ||
|
|
||
| disable_existing_loggers: false |
There was a problem hiding this comment.
Add a new line character at the end of the file.
The file lacks a new line character at the end, which is a best practice for text files.
+ Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # python logging configuration for tasks | |
| version: 1 | |
| formatters: | |
| simple: | |
| format: '[%(asctime)s][%(levelname)s] - %(message)s' | |
| datefmt: '%Y-%m-%d %H:%M:%S' | |
| handlers: | |
| console: | |
| class: logging.StreamHandler | |
| formatter: simple | |
| stream: ext://sys.stdout | |
| file: | |
| class: logging.FileHandler | |
| formatter: simple | |
| # absolute file path | |
| filename: ${hydra.runtime.output_dir}/${now:%Y-%m-%d_%H-%M-%S}-eval.log | |
| mode: w | |
| root: | |
| level: INFO | |
| handlers: [console, file] | |
| disable_existing_loggers: false | |
| # python logging configuration for tasks | |
| version: 1 | |
| formatters: | |
| simple: | |
| format: '[%(asctime)s][%(levelname)s] - %(message)s' | |
| datefmt: '%Y-%m-%d %H:%M:%S' | |
| handlers: | |
| console: | |
| class: logging.StreamHandler | |
| formatter: simple | |
| stream: ext://sys.stdout | |
| file: | |
| class: logging.FileHandler | |
| formatter: simple | |
| # absolute file path | |
| filename: ${hydra.runtime.output_dir}/${now:%Y-%m-%d_%H-%M-%S}-eval.log | |
| mode: w | |
| root: | |
| level: INFO | |
| handlers: [console, file] | |
| disable_existing_loggers: false | |
Tools
yamllint
[error] 22-22: no new line character at the end of file
(new-line-at-end-of-file)
| # python logging configuration for tasks | ||
| version: 1 | ||
| formatters: | ||
| simple: | ||
| format: '[%(asctime)s][%(levelname)s][r${oc.env:LOCAL_RANK}] - %(message)s' | ||
| datefmt: '%Y-%m-%d %H:%M:%S' | ||
| handlers: | ||
| console: | ||
| class: logging.StreamHandler | ||
| formatter: simple | ||
| stream: ext://sys.stdout | ||
| file: | ||
| class: logging.FileHandler | ||
| formatter: simple | ||
| # absolute file path | ||
| filename: ${hydra.runtime.output_dir}/${now:%Y-%m-%d_%H-%M-%S}-rank${oc.env:LOCAL_RANK}.log | ||
| mode: w | ||
| root: | ||
| level: INFO | ||
| handlers: [console, file] | ||
|
|
||
| disable_existing_loggers: false |
There was a problem hiding this comment.
Add a new line character at the end of the file.
The file lacks a new line character at the end, which is a best practice for text files.
+ Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| # python logging configuration for tasks | |
| version: 1 | |
| formatters: | |
| simple: | |
| format: '[%(asctime)s][%(levelname)s][r${oc.env:LOCAL_RANK}] - %(message)s' | |
| datefmt: '%Y-%m-%d %H:%M:%S' | |
| handlers: | |
| console: | |
| class: logging.StreamHandler | |
| formatter: simple | |
| stream: ext://sys.stdout | |
| file: | |
| class: logging.FileHandler | |
| formatter: simple | |
| # absolute file path | |
| filename: ${hydra.runtime.output_dir}/${now:%Y-%m-%d_%H-%M-%S}-rank${oc.env:LOCAL_RANK}.log | |
| mode: w | |
| root: | |
| level: INFO | |
| handlers: [console, file] | |
| disable_existing_loggers: false | |
| # python logging configuration for tasks | |
| version: 1 | |
| formatters: | |
| simple: | |
| format: '[%(asctime)s][%(levelname)s][r${oc.env:LOCAL_RANK}] - %(message)s' | |
| datefmt: '%Y-%m-%d %H:%M:%S' | |
| handlers: | |
| console: | |
| class: logging.StreamHandler | |
| formatter: simple | |
| stream: ext://sys.stdout | |
| file: | |
| class: logging.FileHandler | |
| formatter: simple | |
| # absolute file path | |
| filename: ${hydra.runtime.output_dir}/${now:%Y-%m-%d_%H-%M-%S}-rank${oc.env:LOCAL_RANK}.log | |
| mode: w | |
| root: | |
| level: INFO | |
| handlers: [console, file] | |
| disable_existing_loggers: false | |
Tools
yamllint
[error] 22-22: no new line character at the end of file
(new-line-at-end-of-file)
| def init_context(context): | ||
| context.logger.info("Init context... 0%") | ||
| model = ModelHandler() | ||
| context.user_data.model = model | ||
| context.logger.info("Init context...100%") |
There was a problem hiding this comment.
Consider handling exceptions during model initialization.
Add exception handling to manage potential errors during the model initialization.
def init_context(context):
context.logger.info("Init context... 0%")
try:
model = ModelHandler()
context.user_data.model = model
context.logger.info("Init context...100%")
except Exception as e:
context.logger.error(f"Failed to initialize model: {e}")
raiseCommittable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def init_context(context): | |
| context.logger.info("Init context... 0%") | |
| model = ModelHandler() | |
| context.user_data.model = model | |
| context.logger.info("Init context...100%") | |
| def init_context(context): | |
| context.logger.info("Init context... 0%") | |
| try: | |
| model = ModelHandler() | |
| context.user_data.model = model | |
| context.logger.info("Init context...100%") | |
| except Exception as e: | |
| context.logger.error(f"Failed to initialize model: {e}") | |
| raise |
| def handler(context, event): | ||
| context.logger.info("Run cutie model") | ||
| data = event.body | ||
| buf = io.BytesIO(base64.b64decode(data["image"])) | ||
| shapes = data.get("shapes") | ||
| states = data.get("states") | ||
| image = Image.open(buf).convert("RGB") | ||
| image = np.asarray(image) | ||
|
|
||
| results = { | ||
| "shapes": [], | ||
| "states": [] | ||
| } | ||
|
|
||
| for i, shape in enumerate(shapes): | ||
| shape, state = context.user_data.model.handle(image, shape, states[i] if i<len(states) else None) | ||
| results["shapes"].append(shape) | ||
| results["states"].append(state) | ||
|
|
||
| return context.Response( | ||
| body=json.dumps(results), | ||
| headers={}, | ||
| content_type='application/json', | ||
| status_code=200 | ||
| ) |
There was a problem hiding this comment.
Consider handling exceptions during the handler execution.
Add exception handling to manage potential errors during the handler execution.
def handler(context, event):
context.logger.info("Run cutie model")
try:
data = event.body
buf = io.BytesIO(base64.b64decode(data["image"]))
shapes = data.get("shapes")
states = data.get("states")
image = Image.open(buf).convert("RGB")
image = np.asarray(image)
results = {
"shapes": [],
"states": []
}
for i, shape in enumerate(shapes):
shape, state = context.user_data.model.handle(image, shape, states[i] if i<len(states) else None)
results["shapes"].append(shape)
results["states"].append(state)
return context.Response(
body=json.dumps(results),
headers={},
content_type='application/json',
status_code=200
)
except Exception as e:
context.logger.error(f"Handler execution failed: {e}")
return context.Response(
body=json.dumps({"error": str(e)}),
headers={},
content_type='application/json',
status_code=500
)Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| def handler(context, event): | |
| context.logger.info("Run cutie model") | |
| data = event.body | |
| buf = io.BytesIO(base64.b64decode(data["image"])) | |
| shapes = data.get("shapes") | |
| states = data.get("states") | |
| image = Image.open(buf).convert("RGB") | |
| image = np.asarray(image) | |
| results = { | |
| "shapes": [], | |
| "states": [] | |
| } | |
| for i, shape in enumerate(shapes): | |
| shape, state = context.user_data.model.handle(image, shape, states[i] if i<len(states) else None) | |
| results["shapes"].append(shape) | |
| results["states"].append(state) | |
| return context.Response( | |
| body=json.dumps(results), | |
| headers={}, | |
| content_type='application/json', | |
| status_code=200 | |
| ) | |
| def handler(context, event): | |
| context.logger.info("Run cutie model") | |
| try: | |
| data = event.body | |
| buf = io.BytesIO(base64.b64decode(data["image"])) | |
| shapes = data.get("shapes") | |
| states = data.get("states") | |
| image = Image.open(buf).convert("RGB") | |
| image = np.asarray(image) | |
| results = { | |
| "shapes": [], | |
| "states": [] | |
| } | |
| for i, shape in enumerate(shapes): | |
| shape, state = context.user_data.model.handle(image, shape, states[i] if i<len(states) else None) | |
| results["shapes"].append(shape) | |
| results["states"].append(state) | |
| return context.Response( | |
| body=json.dumps(results), | |
| headers={}, | |
| content_type='application/json', | |
| status_code=200 | |
| ) | |
| except Exception as e: | |
| context.logger.error(f"Handler execution failed: {e}") | |
| return context.Response( | |
| body=json.dumps({"error": str(e)}), | |
| headers={}, | |
| content_type='application/json', | |
| status_code=500 | |
| ) |
| class ModelHandler: | ||
| def __init__(self) -> CUTIE: | ||
| self.device = 'cuda' if torch.cuda.is_available() else 'cpu' | ||
| coco_lvis_weights = '/opt/nuclio/cutie/coco_lvis_h18_itermask.pth' | ||
| cutie_base_weights = '/opt/nuclio/cutie/cutie-base-mega.pth' | ||
|
|
||
| assert os.path.exists(cutie_base_weights), f"{cutie_base_weights} does not exist" | ||
|
|
||
| # load configurations | ||
| initialize(version_base='1.3.2', config_path="./config", job_name="eval_config") | ||
| cfg = compose(config_name="eval_config") | ||
| with open_dict(cfg): | ||
| cfg['weights'] = cutie_base_weights | ||
| get_dataset_cfg(cfg) | ||
|
|
||
| # load model | ||
| cutie = CUTIE(cfg).to(self.device).eval() | ||
| model_weights = torch.load(cutie_base_weights, map_location=torch.device(self.device)) | ||
| cutie.load_weights(model_weights) | ||
|
|
||
| # use one processor per video | ||
| # self.processor = InferenceCore(cutie, cfg=cutie.cfg) | ||
| # self.processor.max_internal_size = 480 | ||
| self.cutie = cutie | ||
|
|
||
| def encode_state(self, state): | ||
| # state.pop('net', None) | ||
|
|
||
| for k,v in state.items(): | ||
| state[k] = jsonpickle.encode(v) | ||
|
|
||
| return state | ||
|
|
||
| def decode_state(self, state): | ||
| for k,v in state.items(): | ||
| state[k] = jsonpickle.decode(v) | ||
|
|
||
| # state['net'] = copy(self.cutie) | ||
|
|
||
| self.cutie = state['net'] | ||
| self.processor = InferenceCore(self.cutie, cfg=self.cutie.cfg) | ||
| self.processor.max_internal_size = 480 | ||
|
|
||
| def handle(self, image: np.array, shape: Optional[List[float]]=None, state: Optional[Dict]=None)->Tuple[List[float], Optional[Dict]]: | ||
| image = prepare_numpy_image(image, self.device) | ||
|
|
||
| if state is None: | ||
| mask = convert_polygon_to_mask(image, shape) # TODO: only handles single object, need to make sure multi-object can be dealt with | ||
| objects = np.unique(mask) | ||
| # background '0' does not count as an object | ||
| objects = objects[objects != 0].tolist() | ||
| mask = torch.from_numpy(mask).to(self.device) | ||
| self.processor = InferenceCore(self.cutie, cfg=self.cutie.cfg) | ||
| self.processor.max_internal_size = 480 | ||
| output_prob = self.processor.step(image, mask, objects=objects) | ||
| state = {} | ||
| state['net'] = self.cutie | ||
| state = self.encode_state(state) | ||
| else: | ||
| self.decode_state(state) | ||
| output_prob = self.processor.step(image) | ||
| state = self.encode_state(state) | ||
|
|
||
| mask = self.processor.output_prob_to_mask(output_prob).cpu().numpy().astype(np.uint8) | ||
|
|
||
| shape = convert_mask_to_polygon(mask)# Convert a mask to a polygon | ||
|
|
||
|
|
||
| return shape, state | ||
|
|
There was a problem hiding this comment.
Consider handling exceptions for model loading and inference.
The ModelHandler class correctly handles the model, but it would be beneficial to add exception handling for model loading and inference to ensure robustness.
try:
model_weights = torch.load(cutie_base_weights, map_location=torch.device(self.device))
cutie.load_weights(model_weights)
except Exception as e:
print(f"Error loading model weights: {e}")
# Handle the exception as needed| restartPolicy: | ||
| name: always | ||
| maximumRetryCount: 3 | ||
| mountMode: volume |
There was a problem hiding this comment.
Add a new line at the end of the file.
The file is missing a new line character at the end.
- mountMode: volume
+Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| mountMode: volume | |
| mountMode: volume | |
|
Motivation and context
In relation to the issue of integrating Video Object Segmentation for Polygon Tracking that has been brought up several times, this pull request proposes following
Related issues are as follows
How has this been tested?
It is still a work in progress. I am new to cvat development and looking for help/suggestion in the following
tools-control.tsxto send the shapes to tracker?Checklist
developbranch(cvat-canvas,
cvat-core,
cvat-data and
cvat-ui)
License
Feel free to contact the maintainers if that's a concern.
Summary by CodeRabbit
New Features
Enhancements
Bug Fixes