Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Expose TRT preview features as EP option #24212

Merged
merged 10 commits into from
Apr 3, 2025

Conversation

toothache
Copy link
Member

Description

Expose TRT preview features as EP option.

Motivation and Context

Add support to turn on TensorRT preview features. For example,

If the IPluginV3OneBuildV2 build capability is used, the plugin can also communicate to TensorRT that certain input-output pairs are aliased (share the same data buffer). TensorRT will query IPluginV3OneBuildV2::getAliasedInput to determine any such aliasing behavior. To use this feature, PreviewFeature::kALIASED_PLUGIN_IO_10_03 must be enabled.

@jywu-msft jywu-msft requested review from chilo-ms and yf711 March 27, 2025 19:30
@tianleiwu tianleiwu marked this pull request as ready for review March 28, 2025 02:54
@tianleiwu tianleiwu closed this Mar 31, 2025
@tianleiwu tianleiwu reopened this Mar 31, 2025
@tianleiwu
Copy link
Contributor

/azp run Windows ARM64 QNN CI Pipeline,Windows x64 QNN CI Pipeline,Windows GPU Doc Gen CI Pipeline,ONNX Runtime Web CI Pipeline,Win_TRT_Minimal_CUDA_Test_CI,Linux CPU CI Pipeline,Linux CPU Minimal Build E2E CI Pipeline,Linux GPU CI Pipeline,Linux GPU TensorRT CI Pipeline,Linux OpenVINO CI Pipeline

@tianleiwu
Copy link
Contributor

/azp run Linux QNN CI Pipeline,onnxruntime-binary-size-checks-ci-pipeline,Big Models,Linux Android Emulator QNN CI Pipeline,Android CI Pipeline,iOS CI Pipeline,ONNX Runtime React Native CI Pipeline,Linux DNNL CI Pipeline,Linux MIGraphX CI Pipeline,Linux ROCm CI Pipeline

Copy link

Azure Pipelines successfully started running 7 pipeline(s).

1 similar comment
Copy link

Azure Pipelines successfully started running 7 pipeline(s).

@toothache
Copy link
Member Author

Hi, @snnn . Do you know why Linux CI pipelines are all failed at Build Docker Image step? Thanks!

#2 [internal] load metadata for onnxruntimebuildcache.azurecr.io/internal/azureml/onnxruntime/build/cpu_x64_ubi8_gcc14:20250124.1
#2 ERROR: failed to authorize: failed to fetch anonymous token: unexpected status from GET request to https://onnxruntimebuildcache.azurecr.io/oauth2/token?scope=repository%3Ainternal%2Fazureml%2Fonnxruntime%2Fbuild%2Fcpu_x64_ubi8_gcc14%3Apull&service=onnxruntimebuildcache.azurecr.io: 401 Unauthorized

@chilo-ms
Copy link
Contributor

chilo-ms commented Apr 1, 2025

Please also help add delete of trt_preview_features pointer in release function https://github.com/microsoft/onnxruntime/blob/main/onnxruntime/core/session/provider_bridge_ort.cc#L2596

ORT_API(void, OrtApis::ReleaseTensorRTProviderOptions, _Frees_ptr_opt_ OrtTensorRTProviderOptionsV2* ptr) {
#ifdef USE_TENSORRT
  if (ptr != nullptr) {
    delete[] ptr->trt_int8_calibration_table_name;
    delete[] ptr->trt_engine_cache_path;
    ...
    + delete[] ptr->trt_preview_features;
   }

@toothache toothache closed this Apr 1, 2025
@toothache toothache reopened this Apr 1, 2025
@toothache
Copy link
Member Author

@chilo-ms some pr gates failed due to permission errors, should I be concerned about it?

@chilo-ms
Copy link
Contributor

chilo-ms commented Apr 1, 2025

@chilo-ms some pr gates failed due to permission errors, should I be concerned about it?

We are working on fixing the permission errors for some CIs.
For the MacOS CI, it seems there is a fix recently in main, you might need to merge main.

@chilo-ms
Copy link
Contributor

chilo-ms commented Apr 2, 2025

/azp run Big Models, Linux CPU Minimal Build E2E CI Pipeline, Linux QNN CI Pipeline, ONNX Runtime Web CI Pipeline, Win_TRT_Minimal_CUDA_Test_CI, Windows ARM64 QNN CI Pipeline, Windows GPU Doc Gen CI Pipeline, Windows x64 QNN CI Pipeline

Copy link

Azure Pipelines successfully started running 7 pipeline(s).

@chilo-ms chilo-ms merged commit 21db38c into microsoft:main Apr 3, 2025
61 of 73 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants