[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

v2.15.0 docker image print error messages like "Unable to register cuDNN/cuFFT... factory " #62987

Open
fancyerii opened this issue Feb 18, 2024 · 6 comments
Assignees
Labels
stat:awaiting tensorflower Status - Awaiting response from tensorflower subtype: ubuntu/linux Ubuntu/Linux Build/Installation Issues TF 2.15 For issues related to 2.15.x type:bug Bug type:build/install Build and install issues

Comments

@fancyerii
Copy link

Issue type

Bug

Have you reproduced the bug with TensorFlow Nightly?

No

Source

binary

TensorFlow version

2.15.0

Custom code

No

OS platform and distribution

ubuntu 18.04

Mobile device

No response

Python version

No response

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

$ docker pull tensorflow/tensorflow:2.15.0-gpu
$ docker run -it --gpus all tensorflow/tensorflow:2.15.0-gpu
$import tensorflow as tf
2024-02-18 02:43:26.283147: E external/local_xla/xla/stream_executor/cuda/cuda_dnn.cc:9261] Unable to register cuDNN factory: Attempting to register factory for plugin cuDNN when one has already been registered
2024-02-18 02:43:26.283204: E external/local_xla/xla/stream_executor/cuda/cuda_fft.cc:607] Unable to register cuFFT factory: Attempting to register factory for plugin cuFFT when one has already been registered
2024-02-18 02:43:26.283926: E external/local_xla/xla/stream_executor/cuda/cuda_blas.cc:1515] Unable to register cuBLAS factory: Attempting to register factory for plugin cuBLAS when one has already been registered
2024-02-18 02:43:26.289011: I tensorflow/core/platform/cpu_feature_guard.cc:182] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
>>> print("Num GPUs Available: ", len(tf.config.list_physical_devices('GPU')))
Num GPUs Available:  8

I tried some examples and it worked. But why this error messages show?

Standalone code to reproduce the issue

Ubuntu 18.04 nvidia-driver 535.104.12

Relevant log output

No response

@google-ml-butler google-ml-butler bot added the type:bug Bug label Feb 18, 2024
@SuryanarayanaY SuryanarayanaY added TF 2.15 For issues related to 2.15.x type:build/install Build and install issues subtype: ubuntu/linux Ubuntu/Linux Build/Installation Issues labels Feb 19, 2024
@SuryanarayanaY
Copy link
Collaborator

Hi @fancyerii ,

Thanks for reporting.Its known issue.It seems this might be due to duplicate registry AFAIK. Though its logging an error it seems the GPU works fine.In your logs also the GPUs also detected and it will not affecting the execution. Could you confirm whether there is a problem with execution of your code?

Same issue was discussed in #62002, #62075.May please refer for more responses there. Thanks!

@SuryanarayanaY
Copy link
Collaborator

CC: @learning-to-play

@SuryanarayanaY SuryanarayanaY added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Feb 19, 2024
@SuryanarayanaY
Copy link
Collaborator

@fancyerii , Just to verify please cross check whether cudnn_version: 8 & cuda_version 12.2 are there or not using the code below.

from tensorflow.python.platform import build_info

print("cudnn_version",build_info.build_info['cudnn_version'])
print("cuda_version",build_info.build_info['cuda_version'])

@amosnier
Copy link
amosnier commented Feb 28, 2024

@SuryanarayanaY , I'm having the same problem as the OP on latest-gpu, just run your recommended cross check and am getting:

>>> print("cudnn_version",build_info.build_info['cudnn_version'])
cudnn_version 8
>>> print("cuda_version",build_info.build_info['cuda_version'])
cuda_version 12.2

So you're saying I just should ignore the "unable to register..." then? They are the only reason I at all went through the trouble of attempting to use Docker images... I'll revert back to my native install then. Thanks.

Note: lines starting with "E" tend to be interpreted as "errors", I guess, i.e. possibly blocking problems. I guess this makes this issue quite a serious one.

@amosnier
Copy link

In fact, there is one more difference for me between native and Docker. With native, I am also getting:

W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

And that seems to be fixed by neither a regular nor a local pip install of tensorrt and friends. So I guess the Docker image still has an added value for me. Unless there is an obvious fix for the local TensorRT problem (I am installing TensorRT according to NVIDIA's instructions as far as I know)?

@rkuo2000
Copy link
rkuo2000 commented Mar 3, 2024

same problem here :(

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
stat:awaiting tensorflower Status - Awaiting response from tensorflower subtype: ubuntu/linux Ubuntu/Linux Build/Installation Issues TF 2.15 For issues related to 2.15.x type:bug Bug type:build/install Build and install issues
Projects
None yet
Development

No branches or pull requests

4 participants