Skip to content

Commit bfac311

Browse files
fengyuentaufengyuentau
committed
Benchmark framework implementation and 3 models added:
* benchmark framework: benchmarks based on configs * added impl and benchmark for YuNet (face detection) * added impl and benchmark for DB (text detection) * added impl and benchmark for CRNN (text recognition)
1 parent af1afb3 commit bfac311

26 files changed

+1649
-4
lines changed

‎.gitignore

+2-3
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,5 @@
11
*.pyc
2-
3-
benchmark/data
4-
benchmark/data/**
2+
**/__pycache__
3+
**/__pycache__/**
54

65
.vscode

‎README.md

+33-1
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,38 @@
22

33
A zoo for models tuned for OpenCV DNN with benchmarks on different platforms.
44

5+
Guidelines:
6+
- To clone this repo, please install [git-lfs](https://git-lfs.github.com/), run `git lfs install` and use `git lfs clone https://github.com/opencv/opencv_zoo`.
7+
- To run benchmark on your hardware settings, please refer to [benchmark/README](./benchmark/README.md).
8+
9+
## Models & Benchmarks
10+
11+
Hardware Setup:
12+
- `CPU x86_64`: INTEL CPU i7-5930K @ 3.50GHz, 6 cores, 12 threads.
13+
- `CPU ARM`: Raspberry 4B, BCM2711B0 @ 1.5GHz (Cortex A-72), 4 cores, 4 threads.
14+
<!--
15+
- `GPU CUDA`: NVIDIA Jetson Nano B01, 128-core Maxwell, Quad-core ARM A57 @ 1.43 GHz.
16+
-->
17+
18+
***Important Notes***:
19+
- The time data that shown on the following tables presents the time elapsed from preprocess (resize is excluded), to a forward pass of a network, and postprocess to get final results.
20+
- The time data that shown on the following tables is averaged from a 100-time run.
21+
- View [benchmark/config](./benchmark/config) for more details on benchmarking different models.
22+
23+
<!--
24+
| Model | Input Size | CPU x86_64 (ms) | CPU ARM (ms) | GPU CUDA (ms) |
25+
|-------|------------|-----------------|--------------|---------------|
26+
| [YuNet](./models/face_detection_yunet) | 160x120 | 2.17 | 8.87 | 14.95 |
27+
| [DB](./models/text_detection_db) | 640x480 | 148.65 | 2759.88 | 218.25 |
28+
| [CRNN](./models/text_recognition_crnn) | 100x32 | 23.23 | 235.87 | 195.20 |
29+
-->
30+
| Model | Input Size | CPU x86_64 (ms) | CPU ARM (ms) |
31+
|-------|------------|-----------------|--------------|
32+
| [YuNet](./models/face_detection_yunet) | 160x120 | 2.17 | 8.87 |
33+
| [DB](./models/text_detection_db) | 640x480 | 148.65 | 2759.88 |
34+
| [CRNN](./models/text_recognition_crnn) | 100x32 | 23.23 | 235.87 |
35+
36+
537
## License
638

7-
OpenCV Zoo is licensed under the [Apache 2.0 license](./LICENCE). Please refer to the licenses of different models for model weights.
39+
OpenCV Zoo is licensed under the [Apache 2.0 license](./LICENSE). Please refer to licenses of different models.

‎benchmark/README.md

+32
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,32 @@
1+
# OpenCV Zoo Benchmark
2+
3+
Benchmarking different models in the zoo.
4+
5+
Data for benchmarking will be downloaded and loaded in [data](./data) based on given config.
6+
7+
Time is measured from data preprocess (resize is excluded), to a forward pass of a network, and postprocess to get final results. The final time data presented is averaged from a 100-time run.
8+
9+
## Preparation
10+
11+
1. Install `python >= 3.6`.
12+
2. Install dependencies: `pip install -r requirements.txt`.
13+
14+
## Benchmarking
15+
16+
Run the following command to benchmark on a given config:
17+
18+
```shell
19+
PYTHONPATH=.. python benchmark.py --cfg ./config/face_detection_yunet.yaml
20+
```
21+
22+
If you are a Windows user and wants to run in CMD/PowerShell, use this command instead:
23+
```shell
24+
set PYTHONPATH=..
25+
python benchmark.py --cfg ./config/face_detection_yunet.yaml
26+
```
27+
<!--
28+
Omit `--cfg` if you want to benchmark all included models:
29+
```shell
30+
PYTHONPATH=.. python benchmark.py
31+
```
32+
-->

‎benchmark/benchmark.py

+182
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,182 @@
1+
import os
2+
import argparse
3+
4+
import yaml
5+
import tqdm
6+
import numpy as np
7+
import cv2 as cv
8+
9+
from models import MODELS
10+
from download import Downloader
11+
12+
parser = argparse.ArgumentParser("Benchmarks for OpenCV Zoo.")
13+
parser.add_argument('--cfg', '-c', type=str,
14+
help='Benchmarking on the given config.')
15+
args = parser.parse_args()
16+
17+
class Timer:
18+
def __init__(self):
19+
self._tm = cv.TickMeter()
20+
21+
self._time_record = []
22+
self._average_time = 0
23+
self._calls = 0
24+
25+
def start(self):
26+
self._tm.start()
27+
28+
def stop(self):
29+
self._tm.stop()
30+
self._calls += 1
31+
self._time_record.append(self._tm.getTimeMilli())
32+
self._average_time = sum(self._time_record) / self._calls
33+
self._tm.reset()
34+
35+
def reset(self):
36+
self._time_record = []
37+
self._average_time = 0
38+
self._calls = 0
39+
40+
def getAverageTime(self):
41+
return self._average_time
42+
43+
44+
class Benchmark:
45+
def __init__(self, **kwargs):
46+
self._fileList = kwargs.pop('fileList', None)
47+
assert self._fileList, 'fileList cannot be empty'
48+
49+
backend_id = kwargs.pop('backend', 'default')
50+
available_backends = dict(
51+
default=cv.dnn.DNN_BACKEND_DEFAULT,
52+
# halide=cv.dnn.DNN_BACKEND_HALIDE,
53+
# inference_engine=cv.dnn.DNN_BACKEND_INFERENCE_ENGINE,
54+
opencv=cv.dnn.DNN_BACKEND_OPENCV,
55+
# vkcom=cv.dnn.DNN_BACKEND_VKCOM,
56+
cuda=cv.dnn.DNN_BACKEND_CUDA
57+
)
58+
self._backend = available_backends[backend_id]
59+
60+
target_id = kwargs.pop('target', 'cpu')
61+
available_targets = dict(
62+
cpu=cv.dnn.DNN_TARGET_CPU,
63+
# opencl=cv.dnn.DNN_TARGET_OPENCL,
64+
# opencl_fp16=cv.dnn.DNN_TARGET_OPENCL_FP16,
65+
# myriad=cv.dnn.DNN_TARGET_MYRIAD,
66+
# vulkan=cv.dnn.DNN_TARGET_VULKAN,
67+
# fpga=cv.dnn.DNN_TARGET_FPGA,
68+
cuda=cv.dnn.DNN_TARGET_CUDA,
69+
cuda_fp16=cv.dnn.DNN_TARGET_CUDA_FP16,
70+
# hddl=cv.dnn.DNN_TARGET_HDDL
71+
)
72+
self._target = available_targets[target_id]
73+
74+
self._sizes = kwargs.pop('sizes', None)
75+
self._repeat = kwargs.pop('repeat', 100)
76+
self._parentPath = kwargs.pop('parentPath', 'benchmark/data')
77+
self._useGroundTruth = kwargs.pop('useDetectionLabel', False) # If it is enable, 'sizes' will not work
78+
assert (self._sizes and not self._useGroundTruth) or (not self._sizes and self._useGroundTruth), 'If \'useDetectionLabel\' is True, \'sizes\' should not exist.'
79+
80+
self._timer = Timer()
81+
self._benchmark_results = dict.fromkeys(self._fileList, dict())
82+
83+
if self._useGroundTruth:
84+
self.loadLabel()
85+
86+
def loadLabel(self):
87+
self._labels = dict.fromkeys(self._fileList, None)
88+
for imgName in self._fileList:
89+
self._labels[imgName] = np.loadtxt(os.path.join(self._parentPath, '{}.txt'.format(imgName[:-4])))
90+
91+
def run(self, model):
92+
model.setBackend(self._backend)
93+
model.setTarget(self._target)
94+
95+
for imgName in self._fileList:
96+
img = cv.imread(os.path.join(self._parentPath, imgName))
97+
if self._useGroundTruth:
98+
for idx, gt in enumerate(self._labels[imgName]):
99+
self._benchmark_results[imgName]['gt{}'.format(idx)] = self._run(
100+
model,
101+
img,
102+
gt,
103+
pbar_msg=' {}, gt{}'.format(imgName, idx)
104+
)
105+
else:
106+
if self._sizes is None:
107+
h, w, _ = img.shape
108+
model.setInputSize([w, h])
109+
self._benchmark_results[imgName][str([w, h])] = self._run(
110+
model,
111+
img,
112+
pbar_msg=' {}, original size {}'.format(imgName, str([w, h]))
113+
)
114+
else:
115+
for size in self._sizes:
116+
imgResized = cv.resize(img, size)
117+
model.setInputSize(size)
118+
self._benchmark_results[imgName][str(size)] = self._run(
119+
model,
120+
imgResized,
121+
pbar_msg=' {}, size {}'.format(imgName, str(size))
122+
)
123+
124+
def printResults(self):
125+
print(' Results:')
126+
for imgName, results in self._benchmark_results.items():
127+
print(' image: {}'.format(imgName))
128+
total_latency = 0
129+
for key, latency in results.items():
130+
total_latency += latency
131+
print(' {}, latency: {:.4f} ms'.format(key, latency))
132+
print(' Average latency: {:.4f} ms'.format(total_latency / len(results)))
133+
134+
def _run(self, model, *args, **kwargs):
135+
self._timer.reset()
136+
pbar = tqdm.tqdm(range(self._repeat))
137+
for _ in pbar:
138+
pbar.set_description(kwargs.get('pbar_msg', None))
139+
140+
self._timer.start()
141+
results = model.infer(*args)
142+
self._timer.stop()
143+
return self._timer.getAverageTime()
144+
145+
146+
def build_from_cfg(cfg, registery):
147+
obj_name = cfg.pop('name')
148+
obj = registery.get(obj_name)
149+
return obj(**cfg)
150+
151+
def prepend_pythonpath(cfg, key1, key2):
152+
pythonpath = os.environ['PYTHONPATH']
153+
if cfg[key1][key2].startswith('/'):
154+
return
155+
cfg[key1][key2] = os.path.join(pythonpath, cfg[key1][key2])
156+
157+
if __name__ == '__main__':
158+
assert args.cfg.endswith('yaml'), 'Currently support configs of yaml format only.'
159+
with open(args.cfg, 'r') as f:
160+
cfg = yaml.safe_load(f)
161+
162+
# prepend PYTHONPATH to each path
163+
prepend_pythonpath(cfg, key1='Data', key2='parentPath')
164+
prepend_pythonpath(cfg, key1='Benchmark', key2='parentPath')
165+
prepend_pythonpath(cfg, key1='Model', key2='modelPath')
166+
167+
168+
# Download data if not exist
169+
print('Loading data:')
170+
downloader = Downloader(**cfg['Data'])
171+
downloader.get()
172+
173+
# Instantiate benchmarking
174+
benchmark = Benchmark(**cfg['Benchmark'])
175+
176+
# Instantiate model
177+
model = build_from_cfg(cfg=cfg['Model'], registery=MODELS)
178+
179+
# Run benchmarking
180+
print('Benchmarking {}:'.format(model.name))
181+
benchmark.run(model)
182+
benchmark.printResults()

‎benchmark/config/face_detection_yunet.yaml

+28
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,28 @@
1+
Data:
2+
name: "Images for Face Detection"
3+
url: "https://drive.google.com/u/0/uc?id=1lOAliAIeOv4olM65YDzE55kn6XjiX2l6&export=download"
4+
sha: "0ba67a9cfd60f7fdb65cdb7c55a1ce76c1193df1"
5+
filename: "face_detection.zip"
6+
parentPath: "benchmark/data"
7+
8+
Benchmark:
9+
name: "Face Detection Benchmark"
10+
parentPath: "benchmark/data/face_detection"
11+
fileList:
12+
- "group.jpg"
13+
- "concerts.jpg"
14+
- "dance.jpg"
15+
backend: "default"
16+
target: "cpu"
17+
sizes: # [w, h], Omit to run at original scale
18+
- [160, 120]
19+
- [640, 480]
20+
repeat: 100 # default 100
21+
22+
Model:
23+
name: "YuNet"
24+
modelPath: "models/face_detection_yunet/face_detection_yunet.onnx"
25+
confThreshold: 0.6
26+
nmsThreshold: 0.3
27+
topK: 5000
28+
keepTopK: 750

‎benchmark/config/text_detection_db.yaml

+27
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,27 @@
1+
Data:
2+
name: "Images for Text Detection"
3+
url: "https://drive.google.com/u/0/uc?id=1lTQdZUau7ujHBqp0P6M1kccnnJgO-dRj&export=download"
4+
sha: "a40cf095ceb77159ddd2a5902f3b4329696dd866"
5+
filename: "text.zip"
6+
parentPath: "benchmark/data"
7+
8+
Benchmark:
9+
name: "Text Detection Benchmark"
10+
parentPath: "benchmark/data/text"
11+
fileList:
12+
- "1.jpg"
13+
- "2.jpg"
14+
- "3.jpg"
15+
backend: "default"
16+
target: "cpu"
17+
sizes: # [w, h], default original scale
18+
- [640, 480]
19+
repeat: 100
20+
21+
Model:
22+
name: "DB"
23+
modelPath: "models/text_detection_db/text_detection_db.onnx"
24+
binaryThreshold: 0.3
25+
polygonThreshold: 0.5
26+
maxCandidates: 200
27+
unclipRatio: 2.0

‎benchmark/config/text_recognition_crnn.yaml

+22
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,22 @@
1+
Data:
2+
name: "Images for Text Detection"
3+
url: "https://drive.google.com/u/0/uc?id=1lTQdZUau7ujHBqp0P6M1kccnnJgO-dRj&export=download"
4+
sha: "a40cf095ceb77159ddd2a5902f3b4329696dd866"
5+
filename: "text.zip"
6+
parentPath: "benchmark/data"
7+
8+
Benchmark:
9+
name: "Text Recognition Benchmark"
10+
parentPath: "benchmark/data/text"
11+
fileList:
12+
- "1.jpg"
13+
- "2.jpg"
14+
- "3.jpg"
15+
backend: "default"
16+
target: "cpu"
17+
useDetectionLabel: True
18+
repeat: 100
19+
20+
Model:
21+
name: "CRNN"
22+
modelPath: "models/text_recognition_crnn/text_recognition_crnn.onnx"

‎benchmark/data/.gitignore

+2
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,2 @@
1+
*
2+
!.gitignore

0 commit comments

Comments
 (0)