Skip to content

Support load TensorRT V3 plugin #24211

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Apr 3, 2025
Merged

Conversation

toothache
Copy link
Member

@toothache toothache commented Mar 27, 2025

Description

TensorRT V3 plugin is not able to load in TensorRT EP. The change deprecates getPluginCreatorList with getAllCreators to load V1 and V3 plugin creators.

Motivation and Context

Support load TensorRT plugin.

Reference: https://github.com/NVIDIA/TensorRT/blob/8c6d69ddec0b2feff12f55472dc5d55cb6861d53/python/src/infer/pyPlugin.cpp#L2971C1-L2995C6

@toothache
Copy link
Member Author

After the change, V3 plugins can be loaded.

2025-03-27 22:39:40.3063320 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:83 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP][V3ONE] CustomEmbLayerNormPluginDynamic, version : 4
2025-03-27 22:39:40.3124991 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:83 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP][V3ONE] CustomEmbLayerNormPluginDynamic, version : 5
2025-03-27 22:39:40.3163747 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:83 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP][V3ONE] CustomSkipLayerNormPluginDynamic, version : 7
2025-03-27 22:39:40.3199090 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:83 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP][V3ONE] CustomSkipLayerNormPluginDynamic, version : 8
2025-03-27 22:39:40.3239330 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:83 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP][V3ONE] CustomSkipLayerNormPluginDynamic, version : 5
2025-03-27 22:39:40.3275216 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:83 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP][V3ONE] CustomSkipLayerNormPluginDynamic, version : 6
2025-03-27 22:39:40.3346971 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:83 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP][V3ONE] InstanceNormalization_TRT, version : 3
2025-03-27 22:39:40.3379989 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:83 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP][V3ONE] ROIAlign_TRT, version : 2
2025-03-27 22:39:40.3419829 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:83 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP][V3ONE] ScatterElements, version : 2
2025-03-27 22:39:40.3453966 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:79 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP] CaskDeconvV2RunnerWeightsTransformerPlugin, version : 1
2025-03-27 22:39:40.3489028 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:79 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP] CaskDeconvV1RunnerWeightsTransformerPlugin, version : 1
2025-03-27 22:39:40.3525361 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:79 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP] CaskConvolutionRunnerWeightsTransformerPlugin, version : 1
2025-03-27 22:39:40.3567290 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:79 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP] CaskFlattenConvolutionRunnerWeightsTransformerPlugin, version : 1
2025-03-27 22:39:40.3617080 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:79 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP] CaskConvActPoolWeightsTransformerPlugin, version : 1
2025-03-27 22:39:40.3652211 [W:onnxruntime:Default, tensorrt_execution_provider_custom_ops.cc:79 onnxruntime::CreateTensorRTCustomOpDomainList] [TensorRT EP] CaskDepSepConvWeightsTransformerPlugin, version : 1

@toothache
Copy link
Member Author

@jywu-msft , @tianleiwu , may I get a review of this change as well? Thanks!

@chilo-ms
Copy link
Contributor

/azp run Big Models, Linux CPU Minimal Build E2E CI Pipeline, Linux QNN CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline

Copy link

Azure Pipelines successfully started running 9 pipeline(s).

@jywu-msft jywu-msft requested a review from yf711 April 1, 2025 03:16
@chilo-ms chilo-ms closed this Apr 1, 2025
@chilo-ms chilo-ms reopened this Apr 1, 2025
 - `getPluginCreatorList` is deprecated, use `getAllCreators` to load plugin creators;
 - support load v1 and v3 creators.
@chilo-ms
Copy link
Contributor

chilo-ms commented Apr 2, 2025

/azp run Big Models, Linux CPU Minimal Build E2E CI Pipeline, Linux QNN CI Pipeline, ONNX Runtime Web CI Pipeline, onnxruntime-binary-size-checks-ci-pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline

Copy link

Azure Pipelines successfully started running 8 pipeline(s).

@chilo-ms chilo-ms merged commit a14d586 into microsoft:main Apr 3, 2025
60 of 74 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants