Skip to content

Commit a509dd8

Browse files
[Model] Add Paddle3D smoke model (#1766)
* add smoke model * add 3d vis * update code * update doc * mv paddle3d from detection to perception * update result for velocity * update code for CI * add set input data for TRT backend * add serving support for smoke model * update code * update code * update code --------- Co-authored-by: DefTruth <[email protected]>
1 parent 81fbd54 commit a509dd8

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

53 files changed

+2610
-26
lines changed

cmake/opencv.cmake

+1-1
Original file line numberDiff line numberDiff line change
@@ -199,6 +199,6 @@ else()
199199
endif()
200200
find_package(OpenCV REQUIRED PATHS ${OpenCV_DIR} NO_DEFAULT_PATH)
201201
include_directories(${OpenCV_INCLUDE_DIRS})
202-
list(APPEND DEPEND_LIBS opencv_core opencv_video opencv_highgui opencv_imgproc opencv_imgcodecs)
202+
list(APPEND DEPEND_LIBS opencv_core opencv_video opencv_highgui opencv_imgproc opencv_imgcodecs opencv_calib3d opencv_features2d opencv_flann)
203203
endif()
204204
endif()
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
English | [简体中文](detection_result_CN.md)
2+
# PerceptionResult target detection result
3+
4+
The PerceptionResult code is defined in `fastdeploy/vision/common/result.h`, which is used to indicate the detected 3D target: two-dimensional target frame, target frame length, width and height, target category and target confidence, target orientation angle and observation angle etc.
5+
6+
## C++ definition
7+
8+
```c++
9+
fastdeploy::vision::PerceptionResult
10+
```
11+
12+
```c++
13+
struct PerceptionResult {
14+
std::vector<float> scores;
15+
std::vector<int32_t> label_ids;
16+
std::vector<std::array<float, 7>> boxes;
17+
std::vector<std::array<float, 3>> center;
18+
std::vector<float>observation_angle;
19+
std::vector<float>yaw_angle;
20+
std::vector<std::array<float, 3>>velocity;
21+
void Clear();
22+
std::string Str();
23+
};
24+
```
25+
26+
- **scores**: Member variable, indicating the confidence of all detected targets, `scores.size()` indicates the number of detected boxes
27+
- **label_ids**: Member variable, representing all detected target categories, the number of elements is consistent with `scores.size()`
28+
- **boxes**: Member variable, representing the coordinates of all detected target boxes, the number of elements is consistent with `scores.size()`, and each box represents xmin, ymin, xmax, ymax in turn with 7 float values , h, w, l, the coordinates of the upper left and lower right corners and the length, width and height of the 3D box
29+
- **center**: member variable, indicating the center point coordinates of all detected target frames, the number of elements is consistent with `scores.size()`, and each frame uses 3 float values to represent the center point coordinates of the frame in turn
30+
- **observation_angle**: Member variable, indicating the observation angle of the detected frame, and the number of elements is consistent with `scores.size()`
31+
- **yaw_angle**: Member variable, indicating the orientation angle of the detected frame, the number of elements is consistent with `scores.size()`
32+
- **velocity**: Member variable, indicating the velocity of the detected frame, the number of elements is consistent with `scores.size()`
33+
- **Clear()**: member function, used to clear the results stored in the structure
34+
- **Str()**: member function, output the information in the structure as a string (for Debug)
35+
36+
## Python definition
37+
38+
```python
39+
fastdeploy.vision.PerceptionResult
40+
```
41+
42+
- **scores**(list of float): Member variable, indicating the confidence of all detected targets, `scores.size()` indicates the number of detected frames
43+
- **label_ids**(list of int): Member variable, representing all detected target categories, the number of elements is consistent with `scores.size()`
44+
- **boxes**(list of list(float)): Member variable, indicating the coordinates of all detected target boxes, the number of elements is the same as `scores.size()`, and each box is in order of 7 float values Indicates xmin, ymin, xmax, ymax, h, w, l, that is, the coordinates of the upper left and lower right corners and the length, width and height of the 3D box
45+
- **center**(list of list(float)): Member variable, which represents the coordinates of the center points of all detected target frames, the number of elements is the same as `scores.size()`, and each frame is represented by 3 floats The values ​​in turn represent the coordinates of the center point of the box
46+
- **observation_angle**: member variable, indicating the orientation angle of the detected frame, and the number of elements is consistent with `scores.size()`
47+
- **yaw_angle**: Member variable, indicating the orientation angle of the detected frame, the number of elements is consistent with `scores.size()`
48+
- **velocity**: Member variable, indicating the velocity of the detected frame, the number of elements is consistent with `scores.size()`
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
简体中文 | [English](perception_result.md)
2+
# PerceptionResult 目标检测结果
3+
4+
PerceptionResult`fastdeploy/vision/common/result.h`中,用于表明检测出来的3D目标的:二维目标框、目标框长宽高、目标类别和目标置信度、目标朝向角和观测角等。
5+
6+
## C++ 定义
7+
8+
```c++
9+
fastdeploy::vision::PerceptionResult
10+
```
11+
12+
```c++
13+
struct PerceptionResult {
14+
std::vector<float> scores;
15+
std::vector<int32_t> label_ids;
16+
std::vector<std::array<float, 7>> boxes;
17+
std::vector<std::array<float, 3>> center;
18+
std::vector<float>observation_angle;
19+
std::vector<float>yaw_angle;
20+
std::vector<std::array<float, 3>>velocity;
21+
void Clear();
22+
std::string Str();
23+
};
24+
```
25+
26+
- **scores**: 成员变量,表示检测出来的所有目标置信度,`scores.size()`表示检测出来框的个数
27+
- **label_ids**: 成员变量,表示检测出来的所有目标类别,其元素个数与`scores.size()`一致
28+
- **boxes**: 成员变量,表示检测出来的所有目标框坐标,其元素个数与`scores.size()`一致,每个框以7个float数值依次表示xmin, ymin, xmax, ymax,h, w, l, 即左上角和右下角坐标以及3D框的长宽高
29+
- **center**: 成员变量,表示检测出来的所有目标框中心点坐标,其元素个数与`scores.size()`一致,每个框以3个float数值依次表示框中心点坐标
30+
- **observation_angle**: 成员变量,表示检测出来的框的观测角,其元素个数与`scores.size()`一致
31+
- **yaw_angle**: 成员变量,表示检测出来的框的朝向角,其元素个数与`scores.size()`一致
32+
- **velocity**: 成员变量,表示检测出来的框的速度,其元素个数与`scores.size()`一致
33+
- **Clear()**: 成员函数,用于清除结构体中存储的结果
34+
- **Str()**: 成员函数,将结构体中的信息以字符串形式输出(用于Debug)
35+
36+
## Python 定义
37+
38+
```python
39+
fastdeploy.vision.PerceptionResult
40+
```
41+
42+
- **scores**(list of float): 成员变量,表示检测出来的所有目标置信度,`scores.size()`表示检测出来框的个数
43+
- **label_ids**(list of int): 成员变量,表示检测出来的所有目标类别,其元素个数与`scores.size()`一致
44+
- **boxes**(list of list(float)): 成员变量,表示检测出来的所有目标框坐标,其元素个数与`scores.size()`一致,每个框以7个float数值依次表示xmin, ymin, xmax, ymax,h, w, l, 即左上角和右下角坐标以及3D框的长宽高
45+
- **center**(list of list(float)): 成员变量,表示检测出来的所有目标框中心点坐标,其元素个数与`scores.size()`一致,每个框以3个float数值依次表示框中心点坐标
46+
- **observation_angle**: 成员变量,表示检测出来的框的朝向角,其元素个数与`scores.size()`一致
47+
- **yaw_angle**: 成员变量,表示检测出来的框的朝向角,其元素个数与`scores.size()`一致
48+
- **velocity**: 成员变量,表示检测出来的框的速度,其元素个数与`scores.size()`一致
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,11 @@
1+
English | [简体中文](README_CN.md)
2+
3+
# Smoke Ready-to-deploy Model
4+
5+
The Smoke deployment model implements the Smoke model from Paddle3D. For more detailed information about the model, please refer to [Smoke Introduction](https://github.com/PaddlePaddle/Paddle3D/tree/develop/docs/models/smoke)
6+
7+
8+
## Detailed Deployment Documents
9+
10+
- [Python Deployment](python)
11+
- [C++ Deployment](cpp)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,12 @@
1+
[English](README.md) | 简体中文
2+
3+
# Smoke 准备部署模型
4+
5+
Smoke 部署模型实现来自 Paddle3D 的 Smoke 模型,模型相关的更多详细信息可以参考[Smoke 介绍](https://github.com/PaddlePaddle/Paddle3D/tree/develop/docs/models/smoke)
6+
7+
8+
9+
## 详细部署文档
10+
11+
- [Python部署](python)
12+
- [C++部署](cpp)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,14 @@
1+
PROJECT(infer_demo C CXX)
2+
CMAKE_MINIMUM_REQUIRED (VERSION 3.10)
3+
4+
# 指定下载解压后的fastdeploy库路径
5+
option(FASTDEPLOY_INSTALL_DIR "Path of downloaded fastdeploy sdk.")
6+
7+
include(${FASTDEPLOY_INSTALL_DIR}/FastDeploy.cmake)
8+
9+
# 添加FastDeploy依赖头文件
10+
include_directories(${FASTDEPLOY_INCS})
11+
12+
add_executable(infer_demo ${PROJECT_SOURCE_DIR}/infer.cc)
13+
# 添加FastDeploy库依赖
14+
target_link_libraries(infer_demo ${FASTDEPLOY_LIBS})
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
English | [简体中文](README_CN.md)
2+
# Smoke C++ Deployment Example
3+
4+
This directory provides an example of `infer.cc` to quickly complete the deployment of Smoke on CPU/GPU.
5+
6+
Before deployment, the following two steps need to be confirmed
7+
8+
- 1. The hardware and software environment meets the requirements, refer to [FastDeploy environment requirements](../../../../../docs/en/build_and_install/download_prebuilt_libraries.md)
9+
- 2. According to the development environment, download the precompiled deployment library and samples code, refer to [FastDeploy prebuilt library](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
10+
11+
Taking CPU inference on Linux as an example, execute the following command in this directory to complete the compilation test. To support this model, you need to ensure FastDeploy version 1.0.6 or higher (x.x.x>=1.0.6)
12+
13+
```bash
14+
mkdir build
15+
cd build
16+
# Download the FastDeploy precompiled library, users can choose the appropriate version to use in the `FastDeploy precompiled library` mentioned above
17+
wget https://bj.bcebos.com/fastdeploy/release/cpp/fastdeploy-linux-x64-x.x.x.tgz
18+
tar xvf fastdeploy-linux-x64-x.x.x.tgz
19+
cmake .. -DFASTDEPLOY_INSTALL_DIR=${PWD}/fastdeploy-linux-x64-x.x.x
20+
make -j
21+
22+
wget https://bj.bcebos.com/fastdeploy/models/smoke.tar.gz
23+
tar -xf smoke.tar.gz
24+
wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
25+
26+
# CPU
27+
./infer_demo smoke smoke_test.png 0
28+
# GPU
29+
./infer_demo smoke smoke_test.png 1
30+
31+
```
32+
33+
The visual result after running is shown in the figure below
34+
35+
<img width="640" src="https://user-images.githubusercontent.com/30516196/230387825-53ac0a09-4137-4e49-9564-197cbc30ff08.png">
36+
37+
The above commands are only applicable to Linux or MacOS. For the usage of SDK under Windows, please refer to:
38+
- [How to use FastDeploy C++ SDK in Windows](../../../../../docs/en/faq/use_sdk_on_windows.md)
39+
40+
## Smoke C++ interface
41+
42+
### Class Smoke
43+
44+
```c++
45+
fastdeploy::vision::detection::Smoke(
46+
const string& model_file,
47+
const string& params_file,
48+
const string& config_file,
49+
const RuntimeOption& runtime_option = RuntimeOption(),
50+
const ModelFormat& model_format = ModelFormat::PADDLE)
51+
```
52+
53+
Smoke model loading and initialization.
54+
55+
**parameter**
56+
57+
> * **model_file**(str): model file path
58+
> * **params_file**(str): parameter file path
59+
> * **config_file**(str): configuration file path
60+
> * **runtime_option**(RuntimeOption): Backend reasoning configuration, the default is None, that is, the default configuration is used
61+
> * **model_format**(ModelFormat): model format, the default is Paddle format
62+
63+
#### Predict function
64+
65+
> ```c++
66+
> Smoke::Predict(cv::Mat* im, PerceptionResult* result)
67+
> ```
68+
>
69+
> Model prediction interface, the input image directly outputs the detection result.
70+
>
71+
> **parameters**
72+
>
73+
> > * **im**: input image, note that it must be in HWC, BGR format
74+
> > * **result**: Detection result, including the detection frame, the confidence of each frame, PerceptionResult description reference [visual model prediction results](../../../../../docs/api /vision_results/)
75+
76+
77+
- [Model Introduction](../../)
78+
- [Python deployment](../python)
79+
- [Vision Model Prediction Results](../../../../../docs/api/vision_results/)
80+
- [How to switch model inference backend engine](../../../../../docs/en/faq/how_to_change_backend.md)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,80 @@
1+
[English](README.md) | 简体中文
2+
# Smoke C++部署示例
3+
4+
本目录下提供 `infer.cc` 快速完成 Smoke 在 CPU/GPU 上部署的示例。
5+
6+
在部署前,需确认以下两个步骤
7+
8+
- 1. 软硬件环境满足要求,参考[FastDeploy环境要求](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
9+
- 2. 根据开发环境,下载预编译部署库和samples代码,参考[FastDeploy预编译库](../../../../../docs/cn/build_and_install/download_prebuilt_libraries.md)
10+
11+
以Linux上 CPU 推理为例,在本目录执行如下命令即可完成编译测试,支持此模型需保证 FastDeploy 版本1.0.6以上(x.x.x>=1.0.6)
12+
13+
```bash
14+
mkdir build
15+
cd build
16+
# 下载FastDeploy预编译库,用户可在上文提到的`FastDeploy预编译库`中自行选择合适的版本使用
17+
wget https://bj.bcebos.com/fastdeploy/release/cpp/fastdeploy-linux-x64-x.x.x.tgz
18+
tar xvf fastdeploy-linux-x64-x.x.x.tgz
19+
cmake .. -DFASTDEPLOY_INSTALL_DIR=${PWD}/fastdeploy-linux-x64-x.x.x
20+
make -j
21+
22+
wget https://bj.bcebos.com/fastdeploy/models/smoke.tar.gz
23+
tar -xf smoke.tar.gz
24+
wget https://bj.bcebos.com/fastdeploy/models/smoke_test.png
25+
26+
# CPU推理
27+
./infer_demo smoke smoke_test.png 0
28+
# GPU推理
29+
./infer_demo smoke smoke_test.png 1
30+
31+
```
32+
33+
运行完成可视化结果如下图所示
34+
35+
<img width="640" src="https://user-images.githubusercontent.com/30516196/230387825-53ac0a09-4137-4e49-9564-197cbc30ff08.png">
36+
37+
以上命令只适用于Linux或MacOS, Windows下SDK的使用方式请参考:
38+
- [如何在Windows中使用FastDeploy C++ SDK](../../../../../docs/cn/faq/use_sdk_on_windows.md)
39+
40+
## Smoke C++ 接口
41+
42+
### Smoke 类
43+
44+
```c++
45+
fastdeploy::vision::detection::Smoke(
46+
const string& model_file,
47+
const string& params_file,
48+
const string& config_file,
49+
const RuntimeOption& runtime_option = RuntimeOption(),
50+
const ModelFormat& model_format = ModelFormat::PADDLE)
51+
```
52+
53+
Smoke模型加载和初始化。
54+
55+
**参数**
56+
57+
> * **model_file**(str): 模型文件路径
58+
> * **params_file**(str): 参数文件路径
59+
> * **config_file**(str): 配置文件路径
60+
> * **runtime_option**(RuntimeOption): 后端推理配置,默认为None,即采用默认配置
61+
> * **model_format**(ModelFormat): 模型格式,默认为Paddle格式
62+
63+
#### Predict函数
64+
65+
> ```c++
66+
> Smoke::Predict(cv::Mat* im, PerceptionResult* result)
67+
> ```
68+
>
69+
> 模型预测接口,输入图像直接输出检测结果。
70+
>
71+
> **参数**
72+
>
73+
> > * **im**: 输入图像,注意需为HWC,BGR格式
74+
> > * **result**: 检测结果,包括检测框,各个框的置信度, PerceptionResult 说明参考[视觉模型预测结果](../../../../../docs/api/vision_results/)
75+
76+
77+
- [模型介绍](../../)
78+
- [Python部署](../python)
79+
- [视觉模型预测结果](../../../../../docs/api/vision_results/)
80+
- [如何切换模型推理后端引擎](../../../../../docs/cn/faq/how_to_change_backend.md)
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,83 @@
1+
// Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
2+
//
3+
// Licensed under the Apache License, Version 2.0 (the "License");
4+
// you may not use this file except in compliance with the License.
5+
// You may obtain a copy of the License at
6+
//
7+
// http://www.apache.org/licenses/LICENSE-2.0
8+
//
9+
// Unless required by applicable law or agreed to in writing, software
10+
// distributed under the License is distributed on an "AS IS" BASIS,
11+
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12+
// See the License for the specific language governing permissions and
13+
// limitations under the License.
14+
15+
#include "fastdeploy/vision.h"
16+
#ifdef WIN32
17+
const char sep = '\\';
18+
#else
19+
const char sep = '/';
20+
#endif
21+
22+
void InitAndInfer(const std::string& model_dir, const std::string& image_file,
23+
const fastdeploy::RuntimeOption& option) {
24+
auto model_file = model_dir + sep + "smoke.pdmodel";
25+
auto params_file = model_dir + sep + "smoke.pdiparams";
26+
auto config_file = model_dir + sep + "infer_cfg.yml";
27+
fastdeploy::vision::EnableFlyCV();
28+
auto model = fastdeploy::vision::perception::Smoke(
29+
model_file, params_file, config_file, option,
30+
fastdeploy::ModelFormat::PADDLE);
31+
assert(model.Initialized());
32+
33+
auto im = cv::imread(image_file);
34+
35+
fastdeploy::vision::PerceptionResult res;
36+
if (!model.Predict(im, &res)) {
37+
std::cerr << "Failed to predict." << std::endl;
38+
return;
39+
}
40+
std::cout << res.Str() << std::endl;
41+
42+
auto vis_im = fastdeploy::vision::VisPerception(im, res, config_file);
43+
cv::imwrite("vis_result.jpg", vis_im);
44+
std::cout << "Visualized result saved in ./vis_result.jpg" << std::endl;
45+
}
46+
47+
int main(int argc, char* argv[]) {
48+
if (argc < 4) {
49+
std::cout << "Usage: infer_demo path/to/paddle_model"
50+
"path/to/image "
51+
"run_option, "
52+
"e.g ./infer_demo ./smoke ./00000.png 0"
53+
<< std::endl;
54+
std::cout << "The data type of run_option is int, 0: run with cpu; 1: run "
55+
"with gpu; 2: run with paddle-trt"
56+
<< std::endl;
57+
return -1;
58+
}
59+
60+
fastdeploy::RuntimeOption option;
61+
if (std::atoi(argv[3]) == 0) {
62+
option.UseCpu();
63+
} else if (std::atoi(argv[3]) == 1) {
64+
option.UseGpu();
65+
} else if (std::atoi(argv[3]) == 2) {
66+
option.UseGpu();
67+
option.UseTrtBackend();
68+
option.EnablePaddleToTrt();
69+
option.SetTrtInputShape("images", {1, 3, 384, 1280});
70+
option.SetTrtInputShape("down_ratios", {1, 2});
71+
option.SetTrtInputShape("trans_cam_to_img", {1, 3, 3});
72+
option.SetTrtInputData("trans_cam_to_img",
73+
{721.53771973, 0., 609.55932617, 0., 721.53771973,
74+
172.85400391, 0, 0, 1});
75+
option.EnablePaddleTrtCollectShape();
76+
}
77+
option.UsePaddleBackend();
78+
79+
std::string model_dir = argv[1];
80+
std::string test_image = argv[2];
81+
InitAndInfer(model_dir, test_image, option);
82+
return 0;
83+
}

0 commit comments

Comments
 (0)