Unstable inference time

[JasonSloan]

Hi!

I run yolo11 model inference for 1000 times in Tesla T4, but I found the time cost was very unstable.
From the cached records, I found most of the time cost was even and looked normal, but the normal ones were always interleaved with​ a few abnormal ones. For example, of the 1000 times inference records, most of the inference time cost was 2ms per image, but there were a few that cost 70ms per image.

I had tried to set a fixed card frequency but it didnt seem to work.

So can you help me with that? Thanks!

[JasonSloan]

Nobody here?

[lix19937]

Use the same inputs or different inputs ? Or you can use trtexec to profile it .

[JasonSloan]

Use the same inputs or different inputs ? Or you can use trtexec to profile it .

The same inputs.
The behaviour of my inference code is different from the profile of trtexec. And the difference only lies in the several abnormal long inference time while others are the same.

[lix19937]

Set warmup, and data copy not include the latency. Lock the gpu freq.

[JasonSloan]

Set warmup, and data copy not include the latency. Lock the gpu freq.

The warm-up phase had already been integrated into the process. Moreover, the GPU frequency had been set to its maximum level. And data copy latency was not taken into consideration. Only the time of enqueueV3 was recorded, yet these times were unstable.