|
| 1 | +# Demonstration of OpenVINO techniques - Model-division and a simplest-way to support custom layers |
| 2 | + |
| 3 | + |
| 4 | +### Description: |
| 5 | + `Model Optimizer` in Intel(r) OpenVINO(tm) toolkit supports model division function. User can specify the region in the model to convert by specifying entry point and exit point with `--input` and `--output` options respectively. |
| 6 | + The expected usage of those options are: |
| 7 | + - **Excluding unnecessary layers:** Removing non-DL related layers (such as JPEG decode) and layers not required for inferencing (such as accuracy metrics calculation) |
| 8 | + - **Load balancing:** Divide a model into multiple parts and cascade them to get the final inferencing result. Each individual part can be run on different device or different timing. |
| 9 | + - **Access to the intermediate result:** Divide a model and get the intermediate feature data to check the model integrity or for the other purposes. |
| 10 | + - **Exclude non-supprted layers:** Convert the model without OpenVINO non-supprted layers. Divide the model and skip non-supported layers to get the IR models. User needs to perform the equivalent processing for the excluded layers to get the correct result. |
| 11 | + |
| 12 | +This project demonstrates how to divide a DL model, and fill the hole for skipped leyers. |
| 13 | +The project includes Python and C++ implementations of **naive** 2D convolution layer to perform the Conv2D task which was supposed to have done by the skipped layer. This could be a good reference when you need to implement a custom layer function to your project but don't want to develop full-blown OpenVINO custom layers due to some restrictions such as development time. |
| 14 | + |
| 15 | + |
| 16 | + |
| 17 | +### Prerequisites: |
| 18 | +- TensorFlow 2.x |
| 19 | +- OpenVINO 2021.4 (2021.x may work) |
| 20 | + |
| 21 | + |
| 22 | +### How to train the model and create trained model files |
| 23 | +You can train the model by just kicking the `training.py` script. `training.py` will use `keras.datasets.mnist` as the training and validation dataset and train the model, and then save the trained model in `SavedModel` format. |
| 24 | +```sh |
| 25 | +python3 training.py |
| 26 | +``` |
| 27 | + |
| 28 | +### How to convert a TF trained model into OpenVINO IR model format |
| 29 | + `Model Optimizer` in OpenVINO converts TF (savedmodel) model into OpenVINO IR model. |
| 30 | + Here's a set of script to convert the model for you. |
| 31 | + |
| 32 | +|script|description| |
| 33 | +|----|----| |
| 34 | +|convert-normal.sh|Convert entire model and generate single IR model file (no division)| |
| 35 | +|convert-divide.sh|Divide the input model and output 2 IR models. All layers are still contained (no skipped layers)| |
| 36 | +|convert-divide-skip.sh|Divide the input model and skip 'target_conv_layer'| |
| 37 | +- The converted models can be found in `./models` folder. |
| 38 | + |
| 39 | +### Tips to find the correct node name for Model Optimizer |
| 40 | + |
| 41 | +Model optimizer requires **MO internal [networkx](https://networkx.org/) graph node name** to specify `--input` and `--output` nodes. You can modify the model optimizer a bit to have it display the list of networkx node names. Add 3 lines on the very bottom of the code snnipet below and run the model optimizer. |
| 42 | + |
| 43 | +`mo/utils/class_registration.py` |
| 44 | +```python |
| 45 | +def apply_replacements_list(graph: Graph, replacers_order: list): |
| 46 | + """ |
| 47 | + Apply all transformations from replacers_order |
| 48 | + """ |
| 49 | + for i, replacer_cls in enumerate(replacers_order): |
| 50 | + apply_transform( |
| 51 | + graph=graph, |
| 52 | + replacer_cls=replacer_cls, |
| 53 | + curr_transform_num=i, |
| 54 | + num_transforms=len(replacers_order)) |
| 55 | + # Display name of available nodes after the 'loader' stage |
| 56 | + if 'LoadFinish' in str(replacer_cls): |
| 57 | + for node in graph.nodes(): |
| 58 | + print(node) |
| 59 | +``` |
| 60 | + |
| 61 | +### How to infer with the models on OpenVINO |
| 62 | +Several versions of scripts are available for the inference testing. |
| 63 | +|script|description| |
| 64 | +|----|----| |
| 65 | +|inference.py|Use simgle, monolithic IR model and run inference| |
| 66 | +|inference-div.py|Take 2 divided IR models and run inference. 2 models will be cascaded.| |
| 67 | +|inference-skip-python.py|Tak2 2 divided IR models which excluded the 'target_conv_layer'. Program is including a Python version of Conv2D and perform convolution for 'target_conv_layer'. **VERY SLOW.** | |
| 68 | +|inference-skip-cpp.py|Tak2 2 divided IR models which excluded the 'target_conv_layer'. Program imports a Python module written in C++ which includes a C++ version of Conv2D. Reasonably fast.| |
| 69 | + |
| 70 | +### How to run `draw-and-infer.py` demo program |
| 71 | +Here's a simple yet bit fun demo application for MNIST CNN. You can draw a number on the screen by mouse or finger-tip and you'll see the real-time inference result. Right-click will clear the screen for another try. Several versions are available. |
| 72 | +|script|description| |
| 73 | +|----|----| |
| 74 | +|draw-and-infer.py|Use the monolithic IR model| |
| 75 | +|draw-and-infer-div.py|Use divided IR models| |
| 76 | +|draw-and-infer-skip-cpp.py|Use divided IR models which excluded 'target_conv_layer'| |
| 77 | + |
| 78 | + |
| 79 | +### Tested environment |
| 80 | +- Windows 10 + VS2019 + OpenVINO 2021.4 |
| 81 | +- Ubuntu 20.04 + OpenVINO 2021.4 |
0 commit comments