quantified ONNX int8 model transform to engine?

[Akiy071]

Description

Hey,guys! I met a problem when i trying to using a quantified ONNX int8 model to transform to engine file.I don't even know that how to solve.I have already tried searching similar issue or questions.There is my error information:

Here is my yolov8n_int8 onnx graph and next is the code export with onnx :

Environment

python version: 3.11.9
tensorrt version:10.6.0
onnxruntime version:1.20.0
GPU: rtx3090

Maybe there's something wrong with the way I quantify?Anyway,thanks for your reply!

[kevinch-nv]

It looks like some issue with your quantization script - from your screenshot the quantized datatype is INT32 which TRT does not support.

[Akiy071]

It looks like some issue with your quantization script - from your screenshot the quantized datatype is INT32 which TRT does not support.

Thank you for your reply! I couldn't find the exact method to fix that issue based on your answer. Therefore, I tried using dynamic quantization to avoid the INT32 node, but unfortunately, it didn't work.May i ask for a way to fix that?If there's no other way, then I'll have to directly use TRT for my quantization.

[kevinch-nv]

I'm not sure which quantization toolkit you're using so I'm not sure how to help there. If the data was in a proper quantized datatype like INT8/FP8 etc then it should work

[Akiy071]

I'm not sure which quantization toolkit you're using so I'm not sure how to help there. If the data was in a proper quantized datatype like INT8/FP8 etc then it should work

My quantization toolkit uses onnxruntime version 1.20.0. There might be something wrong in my code because I don’t fully understand how it’s supposed to be used.
I truly appreciate your response. I will continue investigating until I resolve this issue.