Running ai_spleen_seg_app inference on CPU?

[swederik]

Hi everyone,

I'm trying to run the examples and having some issues. It seems like torchscript fails to load the model that comes with the AI Spleen Seg Data example. I wonder if it's because I don't have an NVIDIA GPU (I'm trying this on a Macbook), and maybe the model was trained using CUDA.

I get this error:

DEBUG:__main__.AISpleenSegApp:Begin compose
DEBUG:__main__.AISpleenSegApp:End compose
DEBUG:__main__.AISpleenSegApp:Begin run
Going to initiate execution of operator DICOMDataLoaderOperator
Executing operator DICOMDataLoaderOperator (Process ID: 51436, Operator ID: 18939fcb-44ac-4b32-9329-1c5f83d246f6)
Done performing execution of operator DICOMDataLoaderOperator

Going to initiate execution of operator DICOMSeriesSelectorOperator
Executing operator DICOMSeriesSelectorOperator (Process ID: 51436, Operator ID: 7ea4fcfb-399a-46f1-98e4-73736df15273)
Done performing execution of operator DICOMSeriesSelectorOperator

Going to initiate execution of operator DICOMSeriesToVolumeOperator
Executing operator DICOMSeriesToVolumeOperator (Process ID: 51436, Operator ID: e5275132-befe-4ada-9b8b-31cd415ad5c0)
Done performing execution of operator DICOMSeriesToVolumeOperator

Going to initiate execution of operator SpleenSegOperator
Executing operator SpleenSegOperator (Process ID: 51436, Operator ID: 1272eced-c34e-4012-a1c8-87fb662e35f3)
False
cpu
Traceback (most recent call last):
  File "app.py", line 75, in <module>
    app_instance.run()
  File "app.py", line 37, in run
    super().run(*args, **kwargs)
  File "/Users/Erik/opt/anaconda3/lib/python3.8/site-packages/monai/deploy/core/application.py", line 429, in run
    executor_obj.run()
  File "/Users/Erik/opt/anaconda3/lib/python3.8/site-packages/monai/deploy/core/executors/single_process_executor.py", line 125, in run
    op.compute(op_exec_context.input_context, op_exec_context.output_context, op_exec_context)
  File "/Users/Erik/Projects/LNQ/monai-deploy-app-sdk/examples/apps/ai_spleen_seg_app/spleen_seg_operator.py", line 87, in compute
    infer_operator.compute(op_input, op_output, context)
  File "/Users/Erik/opt/anaconda3/lib/python3.8/site-packages/monai/deploy/operators/monai_seg_inference_operator.py", line 180, in compute
    d[self._pred_dataset_key] = sliding_window_inference(
  File "/Users/Erik/opt/anaconda3/lib/python3.8/site-packages/monai/inferers/utils.py", line 130, in sliding_window_inference
    seg_prob = predictor(window_data, *args, **kwargs).to(device)  # batched patch segmentation
  File "/Users/Erik/opt/anaconda3/lib/python3.8/site-packages/monai/deploy/core/models/model.py", line 220, in __call__
    if self.predictor:
  File "/Users/Erik/opt/anaconda3/lib/python3.8/site-packages/monai/deploy/core/models/torch_model.py", line 50, in predictor
    self._predictor = torch.jit.load(self.path).eval()
  File "/Users/Erik/opt/anaconda3/lib/python3.8/site-packages/torch/jit/_serialization.py", line 161, in load
    cpp_module = torch._C.import_ir_module(cu, str(f), map_location, _extra_files)
NotImplementedError: Could not run 'aten::empty_strided' with arguments from the 'CUDA' backend. This could be because the operator doesn't exist for this backend, or was omitted during the selective/custom build process (if using custom build). If you are a Facebook employee using PyTorch on mobile, please visit https://fburl.com/ptmfixes for possible resolutions. 'aten::empty_strided' is only available for these backends: [CPU, Meta, BackendSelect, Named, ADInplaceOrView, AutogradOther, AutogradCPU, AutogradCUDA, AutogradXLA, UNKNOWN_TENSOR_TYPE_ID, AutogradMLC, AutogradHPU, AutogradNestedTensor, AutogradPrivateUse1, AutogradPrivateUse2, AutogradPrivateUse3, Tracer, Autocast, Batched, VmapMode].

CPU: registered at aten/src/ATen/RegisterCPU.cpp:16286 [kernel]
Meta: registered at aten/src/ATen/RegisterMeta.cpp:9460 [kernel]
BackendSelect: registered at aten/src/ATen/RegisterBackendSelect.cpp:609 [kernel]
Named: registered at ../aten/src/ATen/core/NamedRegistrations.cpp:7 [backend fallback]
ADInplaceOrView: fallthrough registered at ../aten/src/ATen/core/VariableFallbackKernel.cpp:60 [backend fallback]
AutogradOther: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:9848 [autograd kernel]
AutogradCPU: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:9848 [autograd kernel]
AutogradCUDA: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:9848 [autograd kernel]
AutogradXLA: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:9848 [autograd kernel]
UNKNOWN_TENSOR_TYPE_ID: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:9848 [autograd kernel]
AutogradMLC: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:9848 [autograd kernel]
AutogradHPU: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:9848 [autograd kernel]
AutogradNestedTensor: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:9848 [autograd kernel]
AutogradPrivateUse1: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:9848 [autograd kernel]
AutogradPrivateUse2: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:9848 [autograd kernel]
AutogradPrivateUse3: registered at ../torch/csrc/autograd/generated/VariableType_0.cpp:9848 [autograd kernel]
Tracer: registered at ../torch/csrc/autograd/generated/TraceType_0.cpp:9750 [kernel]
Autocast: fallthrough registered at ../aten/src/ATen/autocast_mode.cpp:255 [backend fallback]
Batched: registered at ../aten/src/ATen/BatchingRegistrations.cpp:1019 [backend fallback]
VmapMode: fallthrough registered at ../aten/src/ATen/VmapModeRegistrations.cpp:33 [backend fallback]```

It looks like in the Torchscript FAQ, they address the fact that the model may need to be saved twice, once for GPU inference, and one for CPU inference: https://pytorch.org/docs/stable/jit.html#frequently-asked-questions

So I was wondering if that Google Drive link for the Spleen Seg Data needs to be updated to include a CPU version of the model.

Thanks,

Erik

[dbericat]

@MMelQin @gigony @vikashg ^^

[MMelQin]

@swederik @dbericat Sorry I somehow did not get the notification.

Yes, the model was trained and script with the use GPU. Since MONAI has a training example for the Spleen, I can see one can use the MONAI training example to retrain the model in using CPU, script the model as such, and then try to use is in the Deploy app.

[gigony]

Sorry @swederik, I also didn't notify this question.

I created an issue with #166 and pushed PR with #167

Could you please test if the patched version works?

pip install -i https://test.pypi.org/simple/ monai-deploy-app-sdk==0.0.167

[MMelQin]

@swederik, @gigony PyTorch model traced to GPU might not be loaded on CPU (forcefully in torch.jit.load specifying map_location=torch.device("cpu"). Notice that the MedNIST example actually trains, traces, and then use the model. For sure, everything is done on CPU, that would work, once jit.load loads the model to the available device. Not sure that's how you tested the fix.

Point is, you PR may not have actually addressed the issue with the Spleen model (traced on GPU, but being loaded on CPU with re-tracing). Can you test with this model (disabling the NVIDAI GPU card or set not visible)?

[gigony]

Hi @MMelQin, I have tested/verified the fix on both the mednist and the spleen model and it worked.

Even though there might be case to map location to CPU doesn't work, I think it is good to try load to CPU if GPU is not available from SDK side

[MMelQin]

1. Yes, jit.load should use the device = torch.device("cuda" if torch.cuda.is_available() else "cpu") to load model, that resolves the issue of the SDK models forcefully loading model on non-existant GPU. The PR addressed this issue.

2. Whether a model traced on GPU can be loaded onto CPU, is another issue, and its success is not guaranteed with this PR. May work for the Spleen model which is great!

[dbericat]

@swederik could you please verify the above issue and PR solved your issue to close this discussion? Thanks!