Skip to content

Commit 306fbe5

Browse files
zhjunqinzhjunqin
authored
[MMSIG-MMDeploy-557]Support the deployment of 3D object detection algorithm SMOKE on Jetson (#2312)
1 parent 7c7e43b commit 306fbe5

31 files changed

+1148
-70
lines changed

‎configs/mmdet3d/mono-detection/mono-detection_dynamic.py

+17
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
_base_ = ['./mono-detection_static.py']
2+
3+
onnx_config = dict(
4+
dynamic_axes={
5+
'input': {
6+
0: 'batch',
7+
2: 'height',
8+
3: 'width'
9+
},
10+
'cls_score': {
11+
0: 'batch',
12+
},
13+
'bbox_pred': {
14+
0: 'batch',
15+
},
16+
},
17+
input_shape=None)

‎configs/mmdet3d/mono-detection/mono-detection_onnxruntime-fp16_dynamic.py

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
_base_ = [
2+
'./mono-detection_dynamic.py', '../../_base_/backends/onnxruntime-fp16.py'
3+
]

‎configs/mmdet3d/mono-detection/mono-detection_onnxruntime_dynamic.py

+3
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,3 @@
1+
_base_ = [
2+
'./mono-detection_dynamic.py', '../../_base_/backends/onnxruntime.py'
3+
]

‎configs/mmdet3d/mono-detection/mono-detection_static.py

+5
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,5 @@
1+
_base_ = ['../../_base_/onnx_config.py']
2+
codebase_config = dict(
3+
type='mmdet3d', task='MonoDetection', model_type='end2end')
4+
onnx_config = dict(
5+
input_names=['input'], output_names=['cls_score', 'bbox_pred'])

‎configs/mmdet3d/mono-detection/mono-detection_tensorrt_dynamic-320x320-1344x1344.py

+11
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
_base_ = ['./mono-detection_dynamic.py', '../../_base_/backends/tensorrt.py']
2+
backend_config = dict(
3+
common_config=dict(max_workspace_size=1 << 30),
4+
model_inputs=[
5+
dict(
6+
input_shapes=dict(
7+
input=dict(
8+
min_shape=[1, 3, 320, 320],
9+
opt_shape=[1, 3, 384, 1280],
10+
max_shape=[1, 3, 1344, 1344]), ))
11+
])

‎docs/en/01-how-to-build/jetsons.md

+19
Original file line numberDiff line numberDiff line change
@@ -194,6 +194,20 @@ Please install dependencies:
194194
sudo apt-get install protobuf-compiler libprotoc-dev
195195
```
196196

197+
#### Install ONNX Runtime [Optional]
198+
199+
Go to [Jetson_Zoo#ONNX_Runtime](https://elinux.org/Jetson_Zoo#ONNX_Runtime) to find **right version** of onnx runtime. Then download and install package.
200+
201+
For example:
202+
203+
```
204+
# Download pip wheel from location mentioned above
205+
$ wget https://nvidia.box.com/shared/static/jy7nqva7l88mq9i8bw3g3sklzf4kccn2.whl -O onnxruntime_gpu-1.10.0-cp36-cp36m-linux_aarch64.whl
206+
207+
# Install pip wheel
208+
$ pip3 install onnxruntime_gpu-1.10.0-cp36-cp36m-linux_aarch64.whl
209+
```
210+
197211
#### Install h5py and pycuda
198212

199213
Model Converter employs HDF5 to save the calibration data for TensorRT INT8 quantization and needs `pycuda` to copy device memory.
@@ -352,3 +366,8 @@ The above inference is done on a [Seeed reComputer built with Jetson Nano module
352366
1. Set `MAX N` mode and perform `sudo nvpmodel -m 0 && sudo jetson_clocks`.
353367
2. Reduce the number of `pre_top_k` in deploy config file like [mmdet pre_top_k](https://github.com/open-mmlab/mmdeploy/blob/34879e638cc2db511e798a376b9a4b9932660fe1/configs/mmdet/_base_/base_static.py#L13) does, e.g., `1000`.
354368
3. Convert the model again and try SDK demo again.
369+
370+
### FAQ
371+
372+
- Error `error: cannot import name 'ProcessGroup' from 'torch.distributed'.`
373+
- Please try install pytorch 1.11 version from [pytorch-for-jetson](https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048).

‎docs/en/04-supported-codebases/mmdet3d.md

+11-9
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
# MMDetection3d Deployment
22

3-
- [Install mmdet3d](#install-mmdet3d)
4-
- [Convert model](#convert-model)
5-
- [Model inference](#model-inference)
6-
- [Supported models](#supported-models)
3+
- [MMDetection3d Deployment](#mmdetection3d-deployment)
4+
- [Install mmdet3d](#install-mmdet3d)
5+
- [Convert model](#convert-model)
6+
- [Model inference](#model-inference)
7+
- [Supported models](#supported-models)
78

89
______________________________________________________________________
910

@@ -74,10 +75,11 @@ The caller needs to refer to the corresponding [python implementation](../../../
7475

7576
## Supported models
7677

77-
| model | dataset | onnxruntime | openvino | tensorrt\* |
78-
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------: | :---------: | :------: | :--------: |
79-
| [centerpoint](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/centerpoint/centerpoint_pillar02_second_secfpn_head-circlenms_8xb4-cyclic-20e_nus-3d.py) | nuScenes | ✔️ | ✔️ | ✔️ |
80-
| [pointpillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py) | nuScenes | ✔️ | ✔️ | ✔️ |
81-
| [pointpillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py) | KITTI | ✔️ | ✔️ | ✔️ |
78+
| model | task | dataset | onnxruntime | openvino | tensorrt\* |
79+
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------: | :------: | :---------: | :------: | :--------: |
80+
| [centerpoint](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/centerpoint/centerpoint_pillar02_second_secfpn_head-circlenms_8xb4-cyclic-20e_nus-3d.py) | voxel detection | nuScenes | ✔️ | ✔️ | ✔️ |
81+
| [pointpillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py) | voxel detection | nuScenes | ✔️ | ✔️ | ✔️ |
82+
| [pointpillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py) | voxel detection | KITTI | ✔️ | ✔️ | ✔️ |
83+
| [smoke](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/smoke/smoke_dla34_dlaneck_gn-all_4xb8-6x_kitti-mono3d.py) | monocular detection | KITTI | ✔️ | x | ✔️ |
8284

8385
- Make sure trt >= 8.4 for some bug fixed, such as ScatterND, dynamic shape crash and so on.

‎docs/en/05-supported-backends/onnxruntime.md

+16
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
4949

5050
- GPU Version
5151

52+
In X64 GPU:
53+
5254
```bash
5355
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-gpu-1.8.1.tgz
5456

@@ -58,6 +60,20 @@ export ONNXRUNTIME_DIR=$(pwd)
5860
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
5961
```
6062

63+
In Arm GPU:
64+
65+
```bash
66+
# Arm not have 1.8.1 version package
67+
wget https://github.com/microsoft/onnxruntime/releases/download/v1.10.0/onnxruntime-linux-aarch64-1.10.0.tgz
68+
69+
tar -zxvf onnxruntime-linux-aarch64-1.10.0.tgz
70+
cd onnxruntime-linux-aarch64-1.10.0
71+
export ONNXRUNTIME_DIR=$(pwd)
72+
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
73+
```
74+
75+
You can also go to [ONNX Runtime Release](https://github.com/microsoft/onnxruntime/releases) to find corresponding release version package.
76+
6177
### Build on Linux
6278

6379
- CPU Version

‎docs/zh_cn/01-how-to-build/jetsons.md

+18
Original file line numberDiff line numberDiff line change
@@ -177,6 +177,19 @@ conda activate mmdeploy
177177
sudo apt-get install protobuf-compiler libprotoc-dev
178178
```
179179

180+
- 安装 ONNX Runtime [可选]
181+
182+
访问 [Jetson_Zoo#ONNX_Runtime](https://elinux.org/Jetson_Zoo#ONNX_Runtime) 找到对应版本的 ONNX Runtime，然后下载并安装。
183+
184+
示例：
185+
186+
```
187+
# Download pip wheel from location mentioned above
188+
$ wget https://nvidia.box.com/shared/static/jy7nqva7l88mq9i8bw3g3sklzf4kccn2.whl -O onnxruntime_gpu-1.10.0-cp36-cp36m-linux_aarch64.whl
189+
# Install pip wheel
190+
$ pip3 install onnxruntime_gpu-1.10.0-cp36-cp36m-linux_aarch64.whl
191+
```
192+
180193
- 安装 h5py 和 pycuda
181194

182195
Model Converter 使用 HDF5 存储 TensorRT INT8 量化的校准数据；需要 pycuda 拷贝显存
@@ -293,3 +306,8 @@ pip install -v -e .
293306
1. 设置为 `MAX N` 模式并执行 `sudo nvpmodel -m 0 && sudo jetson_clocks`。
294307
2. 效仿 [mmdet pre_top_k](https://github.com/open-mmlab/mmdeploy/blob/34879e638cc2db511e798a376b9a4b9932660fe1/configs/mmdet/_base_/base_static.py#L13)，减少配置文件中 `pre_top_k` 的个数，例如 `1000`。
295308
3. 重新进行模型转换并重新运行 demo。
309+
310+
### FAQ
311+
312+
- 错误 `error: cannot import name 'ProcessGroup' from 'torch.distributed'.`
313+
- 请访问 [pytorch-for-jetson](https://forums.developer.nvidia.com/t/pytorch-for-jetson/72048) 安装 pytorch 1.11 版本。

‎docs/zh_cn/04-supported-codebases/mmdet3d.md

+11-9
Original file line numberDiff line numberDiff line change
@@ -1,9 +1,10 @@
11
# MMDetection3d 模型部署
22

3-
- [安装 mmdet3d](#安装-mmdet3d)
4-
- [模型转换](#模型转换)
5-
- [模型推理](#模型推理)
6-
- [模型支持列表](#模型支持列表)
3+
- [MMDetection3d 模型部署](#mmdetection3d-模型部署)
4+
- [安装 mmdet3d](#安装-mmdet3d)
5+
- [模型转换](#模型转换)
6+
- [模型推理](#模型推理)
7+
- [模型支持列表](#模型支持列表)
78

89
______________________________________________________________________
910

@@ -72,10 +73,11 @@ ls -lah centerpoint
7273

7374
## 模型支持列表
7475

75-
| model | dataset | onnxruntime | openvino | tensorrt\* |
76-
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :------: | :---------: | :------: | :--------: |
77-
| [centerpoint](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/centerpoint/centerpoint_pillar02_second_secfpn_head-circlenms_8xb4-cyclic-20e_nus-3d.py) | nuScenes | ✔️ | ✔️ | ✔️ |
78-
| [pointpillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py) | nuScenes | ✔️ | ✔️ | ✔️ |
79-
| [pointpillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py) | KITTI | ✔️ | ✔️ | ✔️ |
76+
| model | task | dataset | onnxruntime | openvino | tensorrt\* |
77+
| :------------------------------------------------------------------------------------------------------------------------------------------------------------------: | :-----------------: | :------: | :---------: | :------: | :--------: |
78+
| [centerpoint](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/centerpoint/centerpoint_pillar02_second_secfpn_head-circlenms_8xb4-cyclic-20e_nus-3d.py) | voxel detection | nuScenes | ✔️ | ✔️ | ✔️ |
79+
| [pointpillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/pointpillars_hv_secfpn_sbn-all_8xb4-2x_nus-3d.py) | voxel detection | nuScenes | ✔️ | ✔️ | ✔️ |
80+
| [pointpillars](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/pointpillars/pointpillars_hv_secfpn_8xb6-160e_kitti-3d-3class.py) | voxel detection | KITTI | ✔️ | ✔️ | ✔️ |
81+
| [smoke](https://github.com/open-mmlab/mmdetection3d/blob/main/configs/smoke/smoke_dla34_dlaneck_gn-all_4xb8-6x_kitti-mono3d.py) | monocular detection | KITTI | ✔️ | x | ✔️ |
8082

8183
- 考虑到 ScatterND、动态 shape 等已知问题，请确保 trt >= 8.4

‎docs/zh_cn/05-supported-backends/onnxruntime.md

+14
Original file line numberDiff line numberDiff line change
@@ -49,6 +49,8 @@ export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
4949

5050
- GPU 版本
5151

52+
X64 GPU:
53+
5254
```bash
5355
wget https://github.com/microsoft/onnxruntime/releases/download/v1.8.1/onnxruntime-linux-x64-gpu-1.8.1.tgz
5456

@@ -58,6 +60,18 @@ export ONNXRUNTIME_DIR=$(pwd)
5860
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
5961
```
6062

63+
Arm GPU:
64+
65+
```bash
66+
# Arm not have 1.8.1 version package
67+
wget https://github.com/microsoft/onnxruntime/releases/download/v1.10.0/onnxruntime-linux-aarch64-1.10.0.tgz
68+
69+
tar -zxvf onnxruntime-linux-aarch64-1.10.0.tgz
70+
cd onnxruntime-linux-aarch64-1.10.0
71+
export ONNXRUNTIME_DIR=$(pwd)
72+
export LD_LIBRARY_PATH=$ONNXRUNTIME_DIR/lib:$LD_LIBRARY_PATH
73+
```
74+
6175
### 在Linux上构建
6276

6377
- CPU 版本

‎mmdeploy/codebase/mmdet3d/deploy/__init__.py

+8-2
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,11 @@
11
# Copyright (c) OpenMMLab. All rights reserved.
2-
from .voxel_detection import MMDetection3d, VoxelDetection
2+
from .mmdet3d import MMDetection3d
3+
from .mono_detection import MonoDetection
4+
from .mono_detection_model import MonoDetectionModel
5+
from .voxel_detection import VoxelDetection
36
from .voxel_detection_model import VoxelDetectionModel
47

5-
__all__ = ['MMDetection3d', 'VoxelDetection', 'VoxelDetectionModel']
8+
__all__ = [
9+
'MMDetection3d', 'VoxelDetection', 'VoxelDetectionModel', 'MonoDetection',
10+
'MonoDetectionModel'
11+
]

‎mmdeploy/codebase/mmdet3d/deploy/mmdet3d.py

+25
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
# Copyright (c) OpenMMLab. All rights reserved.
2+
from mmengine.registry import Registry
3+
4+
from mmdeploy.codebase.base import CODEBASE, MMCodebase
5+
from mmdeploy.utils import Codebase
6+
7+
MMDET3D_TASK = Registry('mmdet3d_tasks')
8+
9+
10+
@CODEBASE.register_module(Codebase.MMDET3D.value)
11+
class MMDetection3d(MMCodebase):
12+
"""MMDetection3d codebase class."""
13+
14+
task_registry = MMDET3D_TASK
15+
16+
@classmethod
17+
def register_deploy_modules(cls):
18+
import mmdeploy.codebase.mmdet3d.models # noqa: F401
19+
20+
@classmethod
21+
def register_all_modules(cls):
22+
from mmdet3d.utils.setup_env import register_all_modules
23+
24+
cls.register_deploy_modules()
25+
register_all_modules(True)

0 commit comments

Comments
 (0)