[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Performance regression: Tensorflow 2.5.0 #51145

Open
liyinhgqw opened this issue Aug 4, 2021 · 9 comments
Open

Performance regression: Tensorflow 2.5.0 #51145

liyinhgqw opened this issue Aug 4, 2021 · 9 comments
Assignees
@penpornk @chunduriv
Labels
comp:ops OPs related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.9 Issues found in the TF 2.9 release (or RCs) type:performance Performance Issue

Comments

@liyinhgqw
Copy link
Contributor
liyinhgqw commented Aug 4, 2021 •
edited
Loading

Please make sure that this is an issue related to performance of TensorFlow.
As per our
GitHub Policy,
we only address code/doc bugs, performance issues, feature requests and
build/installation issues on GitHub. tag:performance_template

System information

  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow): No
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Centos 7
  • Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device:
  • TensorFlow installed from (source or binary): binary
  • TensorFlow version (use command below): 2.5.0
  • Python version: 3.8.0
  • Bazel version (if compiling from source):
  • GCC/Compiler version (if compiling from source):
  • CUDA/cuDNN version:
  • GPU model and memory:

You can collect some of this information using our environment capture
script
You can also obtain the TensorFlow version with:

  1. TF 1.0: python -c "import tensorflow as tf; print(tf.GIT_VERSION, tf.VERSION)"
  2. TF 2.0: python -c "import tensorflow as tf; print(tf.version.GIT_VERSION, tf.version.VERSION)"

Describe the current behavior
I observed significant performance regression (+20% latency) using TF 2.5.0 compared to TF 2.3.0 on CPU.
One op to highlight is sparse_tensor_dense_matmul .
The cause could be Eigen or change of this kernel.

Describe the expected behavior
Performance on parity with TF 2.3.0.

Standalone code to reproduce the issue
Provide a reproducible test case that is the bare minimum necessary to generate
the problem. If possible, please share a link to Colab/Jupyter/any notebook.

https://colab.research.google.com/drive/1gNaQsRZzhmKYFQGMxOO9vzSR6AFpVMci?usp=sharing
@liyinhgqw liyinhgqw added the type:performance Performance Issue label Aug 4, 2021
@pavanky
Copy link
pavanky commented Aug 4, 2021

Performance

  • 2.3.0: ~245 ms
  • 2.4.0: ~240 ms
  • 2.5.0: ~350ms
  • tf-nightly (latest): ~350ms

Something broke between 2.4 and 2.5

@SohaibAnwaar
Copy link

Hy! I am processing matrix multiplication operations on TensorFlow and NumPy to check the processing time difference on CPU and GPU. It's taking more time to process operation on GPU than on CPU

On Cpu its taking:1.15 sec

On GPU it's taking: 34 sec

why so? According to my understanding GPU should perform fast operations than CPU..

CPU CODE

    import numpy as np
    import time
    from tqdm import tqdm
    
    
    # Making Tensor Constant
    np_arr = np.asarray([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [4.0, 5.0, 6.0]])
    print("Starting cpu processing")
    st = time.time()
    for i in tqdm(range(0,1000000)):
        np_arr @ np_arr
        
    # Took 1 sec    
    print(f"Time Taken BY CPU {time.time() -st}")

GPU Code

    import tensorflow as tf
    physical_devices = tf.config.list_physical_devices('GPU') 
    
    # Gpu Settings for rtx3070
    from tensorflow.compat.v1.keras.backend import set_session
    config = tf.compat.v1.ConfigProto()
    config.gpu_options.allow_growth = True  # dynamically grow the memory used on the GPU
    config.log_device_placement = True  # to log device placement (on which device the operation ran)
    sess = tf.compat.v1.Session(config=config)
    set_session(sess)
    
    import time
    from tqdm import tqdm
    
    print("Here is the tensorflow version ",tf.__version__)
    
    # Making Tensor Constant
    tensor1 = tf.constant([[1.0, 2.0, 3.0], [4.0, 5.0, 6.0], [4.0, 5.0, 6.0]])
    
    # Getting the shape of the tensor
    print(tensor1.get_shape())
    
    
    st = time.time()
    for i in tqdm(range(0,1000000)):
        # Using tensorflow function to use gpu functionality
        tf.linalg.matmul(tensor1, tensor1)
    
    # Took 34 sec
    print(f"Time Taken BY GPU {time.time() -st}")

@sushreebarsa sushreebarsa added TF 2.5 Issues related to TF 2.5 comp:ops OPs related issues labels Aug 6, 2021
@sushreebarsa
Copy link
Contributor
sushreebarsa commented Aug 6, 2021 •
edited
Loading

@rmothukuru Was able to reproduce on colab using TF v2.3,v2.4, v2.5 & tf-nightly,please find the gists attached here.
Thank you!

@sushreebarsa sushreebarsa removed their assignment Aug 6, 2021
@ImtiazSajwani
Copy link

Try using the Intel Tensor Flow 2.5, pip install intel-tensorflow==2.5.0

@penpornk
Copy link
Member

@ImtiazSajwani Intel TensorFlow doesn't have any optimization for SparseTensor ops, so I think the performance will be similar.

@liyinhgqw
Copy link
Contributor Author

@penpornk , do you have a plan to patch the performance fix of Eigen and TF to master or any version of release?

@penpornk
Copy link
Member

@liyinhgqw Currently there is no plan. But I can investigate if the issue can be fixed in the master branch. (The patch for v2.5 alone requires reverting / changing 5 commits between 2.3 and 2.5. There could be more commits after 2.5 which affects compiler optimization.)

@kwlzn
Copy link
kwlzn commented Dec 7, 2021 •
edited
Loading

@penpornk that would be super helpful - thank you!

ideally, we could get a fix for this onto master for the 2.8 release (and future versions). do you think that would be possible?

@chunduriv
Copy link
Contributor

I was able to reproduce the issue in TF2.8 and tf-nightly(2.9.0-dev20220401). Please find the gist here for reference. Thanks!

@chunduriv chunduriv added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Apr 1, 2022
@tilakrayal tilakrayal added TF 2.9 Issues found in the TF 2.9 release (or RCs) and removed TF 2.5 Issues related to TF 2.5 labels Aug 30, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@penpornk penpornk

@chunduriv chunduriv

Labels
comp:ops OPs related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower TF 2.9 Issues found in the TF 2.9 release (or RCs) type:performance Performance Issue
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@pavanky @liyinhgqw @kwlzn @SohaibAnwaar @penpornk @ImtiazSajwani @rmothukuru @chunduriv @tilakrayal @sushreebarsa