Add support for TensorRT 10 #68715
Draft
Add support for TensorRT 10 #68715
+1,096
−131
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: Meenakshi Venkataraman <meenakshiv@nvidia.com>
Signed-off-by: Meenakshi Venkataraman <meenakshiv@nvidia.com>
- TRT10 changed the output dtype of shape layers from int32 to int64, which causes mismatches with other layers. This commit adds cast layers to avoid the mismatches. - Note that this also adds support for the out_dtype attribute of TF's Shape operator.
- Also avoids runtime warnings about the hasImplicitBatchDimension API.
- It seems that the number of inputs no longer needs to be divided by the number of profiles. This manifested as a confusing bug because it caused values in the array of min/max/opt to be written over the top of existing values instead of at the end, and this subsequently prevented shapes from being matched when looking up engines.
- use_dynamic_shape=False is not supported since TensorRT 10.0. - Also expands the related error message.
|
Hm, afaik I should not be the reviewer on this -- it should be someone at Nvidia affiliated with TensorRT. |
|
@beckerhe who can review this PR? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
TensorRT 10 makes a number of breaking changes to its API, requiring fixes in many places in the TF-TRT code. Most of the changes are straightforward though.
Opening as a draft because there is one set of tests that are still known to fail.
cc @nluehr @cliffwoolley