Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature Request] How should I use symmetric quantization to quantify weight and obtain the correct quantization model? #24183

Open
cosonjia opened this issue Mar 26, 2025 · 0 comments
Labels
feature request request for unsupported feature or enhancement quantization issues related to quantization

Comments

@cosonjia
Copy link

Describe the feature request

Describe the bug:

I am able to do quantization with:

quantized_model = quantize_dynamic(
        model_input=model_input,
        model_output=model_output,
        # optimize_model=False,
        weight_type=QuantType.QInt8, 
        # activation_type=QuantType.QInt8,
        # op_types_to_quantize=["Conv", "MatMul", "Add","Relu"], 
        extra_options={"UseSymmetric": True, "ActivationSymmetric": True, 
                       "EnableSubgraph": True, "ForceQuantizeNoInputCheck": True}  
    )

but I got the following error while doing the inference:

File "/home/pc10/anaconda3/envs/evo-venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 465, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "/home/pc10/anaconda3/envs/evo-venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 537, in _create_inference_session
sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for ConvInteger(10) node with name 'Conv__116_quant'

System information

OS Platform and Distribution: Ubuntu Ubuntu 20.04.5 LTS(5.15.0-48-generic)
CPU 12th Gen Intel(R) Core(TM) i9-12900K
ONNX Runtime installed by pip:
ONNX Runtime version: 1.20.1
Python version: 3.11.11

How should I use symmetric quantization to quantify weight and obtain the correct quantization model?

Describe scenario use case

quantized_model = quantize_dynamic(
        model_input=model_input,
        model_output=model_output,
        # optimize_model=False,
        weight_type=QuantType.QInt8, 
        # activation_type=QuantType.QInt8,
        # op_types_to_quantize=["Conv", "MatMul", "Add","Relu"], 
        extra_options={"UseSymmetric": True, "ActivationSymmetric": True, 
                       "EnableSubgraph": True, "ForceQuantizeNoInputCheck": True}  
    )
@cosonjia cosonjia added the feature request request for unsupported feature or enhancement label Mar 26, 2025
@github-actions github-actions bot added the quantization issues related to quantization label Mar 26, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request request for unsupported feature or enhancement quantization issues related to quantization
Projects
None yet
Development

No branches or pull requests

1 participant