Skip to content

[Feature Request] How should I use symmetric quantization to quantify weight and obtain the correct quantization model? #24183

Open
@cosonjia

Description

@cosonjia

Describe the feature request

Describe the bug:

I am able to do quantization with:

quantized_model = quantize_dynamic(
        model_input=model_input,
        model_output=model_output,
        # optimize_model=False,
        weight_type=QuantType.QInt8, 
        # activation_type=QuantType.QInt8,
        # op_types_to_quantize=["Conv", "MatMul", "Add","Relu"], 
        extra_options={"UseSymmetric": True, "ActivationSymmetric": True, 
                       "EnableSubgraph": True, "ForceQuantizeNoInputCheck": True}  
    )

but I got the following error while doing the inference:

File "/home/pc10/anaconda3/envs/evo-venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 465, in init
self._create_inference_session(providers, provider_options, disabled_optimizers)
File "/home/pc10/anaconda3/envs/evo-venv/lib/python3.11/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py", line 537, in _create_inference_session
sess.initialize_session(providers, provider_options, disabled_optimizers)
onnxruntime.capi.onnxruntime_pybind11_state.NotImplemented: [ONNXRuntimeError] : 9 : NOT_IMPLEMENTED : Could not find an implementation for ConvInteger(10) node with name 'Conv__116_quant'

System information

OS Platform and Distribution: Ubuntu Ubuntu 20.04.5 LTS(5.15.0-48-generic)
CPU 12th Gen Intel(R) Core(TM) i9-12900K
ONNX Runtime installed by pip:
ONNX Runtime version: 1.20.1
Python version: 3.11.11

How should I use symmetric quantization to quantify weight and obtain the correct quantization model?

Describe scenario use case

quantized_model = quantize_dynamic(
        model_input=model_input,
        model_output=model_output,
        # optimize_model=False,
        weight_type=QuantType.QInt8, 
        # activation_type=QuantType.QInt8,
        # op_types_to_quantize=["Conv", "MatMul", "Add","Relu"], 
        extra_options={"UseSymmetric": True, "ActivationSymmetric": True, 
                       "EnableSubgraph": True, "ForceQuantizeNoInputCheck": True}  
    )

Metadata

Metadata

Assignees

No one assigned

    Labels

    feature requestrequest for unsupported feature or enhancementquantizationissues related to quantization

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions