Skip to content

Commit b626536

Browse files
committed
comfirmed on win10 + ov2021.4
1 parent 3c884ab commit b626536

9 files changed

+81
-0
lines changed

README.md

Lines changed: 81 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,81 @@
1+
# Demonstration of OpenVINO techniques - Model-division and a simplest-way to support custom layers
2+
3+
4+
### Description:
5+
`Model Optimizer` in Intel(r) OpenVINO(tm) toolkit supports model division function. User can specify the region in the model to convert by specifying entry point and exit point with `--input` and `--output` options respectively.
6+
The expected usage of those options are:
7+
- **Excluding unnecessary layers:** Removing non-DL related layers (such as JPEG decode) and layers not required for inferencing (such as accuracy metrics calculation)
8+
- **Load balancing:** Divide a model into multiple parts and cascade them to get the final inferencing result. Each individual part can be run on different device or different timing.
9+
- **Access to the intermediate result:** Divide a model and get the intermediate feature data to check the model integrity or for the other purposes.
10+
- **Exclude non-supprted layers:** Convert the model without OpenVINO non-supprted layers. Divide the model and skip non-supported layers to get the IR models. User needs to perform the equivalent processing for the excluded layers to get the correct result.
11+
12+
This project demonstrates how to divide a DL model, and fill the hole for skipped leyers.
13+
The project includes Python and C++ implementations of **naive** 2D convolution layer to perform the Conv2D task which was supposed to have done by the skipped layer. This could be a good reference when you need to implement a custom layer function to your project but don't want to develop full-blown OpenVINO custom layers due to some restrictions such as development time.
14+
15+
![image](./resources/model-division.jpg)
16+
17+
### Prerequisites:
18+
- TensorFlow 2.x
19+
- OpenVINO 2021.4 (2021.x may work)
20+
21+
22+
### How to train the model and create trained model files
23+
You can train the model by just kicking the `training.py` script. `training.py` will use `keras.datasets.mnist` as the training and validation dataset and train the model, and then save the trained model in `SavedModel` format.
24+
```sh
25+
python3 training.py
26+
```
27+
28+
### How to convert a TF trained model into OpenVINO IR model format
29+
`Model Optimizer` in OpenVINO converts TF (savedmodel) model into OpenVINO IR model.
30+
Here's a set of script to convert the model for you.
31+
32+
|script|description|
33+
|----|----|
34+
|convert-normal.sh|Convert entire model and generate single IR model file (no division)|
35+
|convert-divide.sh|Divide the input model and output 2 IR models. All layers are still contained (no skipped layers)|
36+
|convert-divide-skip.sh|Divide the input model and skip 'target_conv_layer'|
37+
- The converted models can be found in `./models` folder.
38+
39+
### Tips to find the correct node name for Model Optimizer
40+
41+
Model optimizer requires **MO internal [networkx](https://networkx.org/) graph node name** to specify `--input` and `--output` nodes. You can modify the model optimizer a bit to have it display the list of networkx node names. Add 3 lines on the very bottom of the code snnipet below and run the model optimizer.
42+
43+
`mo/utils/class_registration.py`
44+
```python
45+
def apply_replacements_list(graph: Graph, replacers_order: list):
46+
"""
47+
Apply all transformations from replacers_order
48+
"""
49+
for i, replacer_cls in enumerate(replacers_order):
50+
apply_transform(
51+
graph=graph,
52+
replacer_cls=replacer_cls,
53+
curr_transform_num=i,
54+
num_transforms=len(replacers_order))
55+
# Display name of available nodes after the 'loader' stage
56+
if 'LoadFinish' in str(replacer_cls):
57+
for node in graph.nodes():
58+
print(node)
59+
```
60+
61+
### How to infer with the models on OpenVINO
62+
Several versions of scripts are available for the inference testing.
63+
|script|description|
64+
|----|----|
65+
|inference.py|Use simgle, monolithic IR model and run inference|
66+
|inference-div.py|Take 2 divided IR models and run inference. 2 models will be cascaded.|
67+
|inference-skip-python.py|Tak2 2 divided IR models which excluded the 'target_conv_layer'. Program is including a Python version of Conv2D and perform convolution for 'target_conv_layer'. **VERY SLOW.** |
68+
|inference-skip-cpp.py|Tak2 2 divided IR models which excluded the 'target_conv_layer'. Program imports a Python module written in C++ which includes a C++ version of Conv2D. Reasonably fast.|
69+
70+
### How to run `draw-and-infer.py` demo program
71+
Here's a simple yet bit fun demo application for MNIST CNN. You can draw a number on the screen by mouse or finger-tip and you'll see the real-time inference result. Right-click will clear the screen for another try. Several versions are available.
72+
|script|description|
73+
|----|----|
74+
|draw-and-infer.py|Use the monolithic IR model|
75+
|draw-and-infer-div.py|Use divided IR models|
76+
|draw-and-infer-skip-cpp.py|Use divided IR models which excluded 'target_conv_layer'|
77+
![draw-and-infer](./resources/draw-and-infer.png)
78+
79+
### Tested environment
80+
- Windows 10 + VS2019 + OpenVINO 2021.4
81+
- Ubuntu 20.04 + OpenVINO 2021.4

build.bat

100755100644
File mode changed.

build.sh

100755100644
File mode changed.
File renamed without changes.
File renamed without changes.

convert-divide.bat

100755100644
File mode changed.

convert-divide.sh

100755100644
File mode changed.

convert-normal.bat

100755100644
File mode changed.

convert-normal.sh

100755100644
File mode changed.

0 commit comments

Comments
 (0)