[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TF 2.3 - loading of saved_model from disk with ragged=True input is slower than ragged=False #43263

Open
michalderdak opened this issue Sep 16, 2020 · 5 comments
Open

TF 2.3 - loading of saved_model from disk with ragged=True input is slower than ragged=False #43263

michalderdak opened this issue Sep 16, 2020 · 5 comments
Assignees
@k-w-w
Labels
comp:keras Keras related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.3 Issues related to TF 2.3 type:performance Performance Issue

Comments

@michalderdak
Copy link

System information
Google Colab Notebook with TF 2.3

Describe the current behavior
Loading saved model with input tf.keras.layers.Input(shape=[None], dtype=tf.int64, ragged=True) is 5-10x slower than tf.keras.layers.Input(shape=[None], dtype=tf.int64, ragged=False)

Describe the expected behavior
The above should not impact performance

Standalone code to reproduce the issue
https://colab.research.google.com/drive/10ZXvdbp1lf8X2LLtemcHr2rX692n2jv9?usp=sharing

Other info / logs
The above is achieved with a custom model implementation. However, when I used the model from one of the Tensorflow tutorials I could not replicate the issue.

On top of that, we were able to replicate this issue with tensorflow serving where loading of the saved_model + first prediction was 5-10x slower than model without ragged tensors.
@michalderdak michalderdak added the type:performance Performance Issue label Sep 16, 2020
@amahendrakar
Copy link
Contributor

Was able to reproduce the issue with TF v2.3 and TF-nightly.

No time difference when running TensorFlow example, slows down when running a custom model implementation. Please find the attached gist. Thanks!

@amahendrakar amahendrakar added comp:apis Highlevel API related issues TF 2.3 Issues related to TF 2.3 labels Sep 16, 2020
@amahendrakar amahendrakar assigned ymodak and unassigned amahendrakar Sep 16, 2020
@ymodak ymodak added comp:keras Keras related issues and removed comp:apis Highlevel API related issues labels Sep 18, 2020
@ymodak ymodak assigned tomerk and k-w-w and unassigned ymodak and tomerk Sep 21, 2020
@tanguycdls tanguycdls mentioned this issue Sep 28, 2020
Closed
@tanguycdls
Copy link

@michalderdak did you have any luck on this ? we're still struggling with it in our side... thanks

@chunduriv
Copy link
Contributor

@michalderdak, Sorry for the late response. Is this still an issue for you?

Can you please try recent TF2.6 or tf-nightly and let us know whether it is persisting? Thanks!

@chunduriv chunduriv self-assigned this Oct 27, 2021
@chunduriv chunduriv added the stat:awaiting response Status - Awaiting response from author label Oct 27, 2021
@michalderdak
Copy link
Author

@chunduriv Thanks for the reply! We have gone with another alternative that does not use ragged tensors, however, I will test with TF2.6 and let you know if I can replicate the issue

@tensorflowbutler tensorflowbutler removed the stat:awaiting response Status - Awaiting response from author label Oct 30, 2021
@michalderdak
Copy link
Author

@chunduriv I've replicated the issue with TF2.6 and tf-nightly. Furthermore, I have added examples without our custom loss function to have as little custom code as possible.

see the notebooks here:
https://colab.research.google.com/drive/1FW2cVAGSL5aWuFBbDcLNbwvtseOaOYO2?usp=sharing
https://colab.research.google.com/drive/1YNKSG21CQf3Q6mQFzrLKNO6FzB7Z6j2T?usp=sharing

@chunduriv chunduriv added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Nov 8, 2021
@chunduriv chunduriv removed their assignment Nov 8, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@k-w-w k-w-w

Labels
comp:keras Keras related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.3 Issues related to TF 2.3 type:performance Performance Issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@tomerk @michalderdak @tanguycdls @tensorflowbutler @k-w-w @ymodak @amahendrakar @chunduriv