Understanding `Could not find TensorRT` #62976

tymorrow · 2024-02-16T01:50:04Z

Issue type

Others

Have you reproduced the bug with TensorFlow Nightly?

Yes

Source

binary

TensorFlow version

tf 2.12, 2.15, nightly

Custom code

No

OS platform and distribution

Linux Ubuntu 22.04, MacOS 12, Windows 2022

Mobile device

No response

Python version

3.8 - 3.10

Bazel version

No response

GCC/compiler version

No response

CUDA/cuDNN version

No response

GPU model and memory

No response

Current behavior?

A warning "Could not find TensorRT" is produced when importing TensorFlow on Ubuntu 22.04.
However, this warning does not appear on Windows 2022 and MacOS 12.

	MacOS12	Windows 2022	Ubuntu 22.04
TF 2.12 w/ Python 3.8	no TensorRT warning	no TensorRT warning	TensorRT warning
TF 2.12 w/ Python 3.9	no TensorRT warning	no TensorRT warning	TensorRT warning
TF 2.12 w/ Python 3.10	no TensorRT warning	no TensorRT warning	TensorRT warning
TF 2.15 w/ Python 3.10	not checked	not checked	TensorRT warning
TF nightly w/ Python 3.10	not checked	not checked	TensorRT warning

I would expect the behavior to be consistent across platforms unless their is an Ubuntu-specific reason.
This issue is intended to help me understand:

Why is there this inconsistency?
Can it either be made consistent or can the warning be updated to explain the platform-specific benefits behind why TensorRT should be installed?

Additional remarks:

I am not trying to utilize TensorRT, so installing an additional package to satisfy this message is not a desired fix, but it may be for others.
I am also not trying to suppress logging warnings, just trying to understand the purpose of it only appearing on Ubuntu.

Thank you for your time!

Standalone code to reproduce the issue

import tensorflow

Relevant log output

W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT

sushreebarsa · 2024-02-19T10:03:09Z

@tymorrow TensorRT installation might require additional steps or dependencies, particularly on specific platforms like Ubuntu. TensorRT is an NVIDIA-specific library optimized for NVIDIA GPUs. It might not be available on non-NVIDIA hardware or specific Linux distributions like Ubuntu due to package management differences.
Thank you!

tymorrow · 2024-02-21T00:02:07Z

Hi @sushreebarsa, could you elaborate on why the warning does not appear on Windows or MacOS?

I'm also wondering why the warning appears at all; is it specifically to tell me what you just said?
If so, could it instead say what you just said to be more informative?

Thank you for your time.

zeke-john · 2024-04-22T00:37:27Z

Any update? Been getting the same issue for about a month now.

This what i am getting:

2024-04-21 08:00:13.414353: I tensorflow/core/util/port.cc:113] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2024-04-21 08:00:13.552935: I tensorflow/core/platform/cpu_feature_guard.cc:210] This TensorFlow binary is optimized to use available CPU instructions in performance-critical operations.
To enable the following instructions: AVX2 AVX512F AVX512_VNNI FMA, in other operations, rebuild TensorFlow with the appropriate compiler flags.
2024-04-21 08:00:14.655097: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Could not find TensorRT
2024-04-21 08:00:16.735268: I external/local_xla/xla/stream_executor/cuda/cuda_executor.cc:984] could not open file to read NUMA node: /sys/bus/pci/devices/0000:17:00.0/numa_node
Your kernel may have been built without NUMA support.
2024-04-21 08:00:16.828498: W tensorflow/core/common_runtime/gpu/gpu_device.cc:2251] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.

I have manually installed TensorRT and still have the same error

Here is my Dockerfile for reference:

FROM nvidia/cuda:12.0.0-devel-ubuntu20.04

ENV DEBIAN_FRONTEND noninteractive

RUN apt-get update && \
    apt-get install -y git ffmpeg libssl-dev lsb-release wget curl


ENV PATH="/root/miniconda3/bin:${PATH}"
ARG PATH="/root/miniconda3/bin:${PATH}"

RUN wget \
    https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh \
    && mkdir /root/.conda \
    && bash Miniconda3-latest-Linux-x86_64.sh -b \
    && rm -f Miniconda3-latest-Linux-x86_64.sh s

RUN conda --version
# conda 24.1.2


# CUDNN
RUN apt clean
RUN apt update 
RUN apt autoremove

RUN wget https://developer.download.nvidia.com/compute/cudnn/9.1.0/local_installers/cudnn-local-repo-ubuntu2004-9.1.0_1.0-1_amd64.deb
RUN dpkg -i cudnn-local-repo-ubuntu2004-9.1.0_1.0-1_amd64.deb
RUN cp /var/cudnn-local-repo-ubuntu2004-9.1.0/cudnn-*-keyring.gpg /usr/share/keyrings/
RUN apt-get update
RUN apt-get -y install cudnn-cuda-12


# CUBLASS
RUN curl https://developer.download.nvidia.com/hpc-sdk/ubuntu/DEB-GPG-KEY-NVIDIA-HPC-SDK |  gpg --dearmor -o /usr/share/keyrings/nvidia-hpcsdk-archive-keyring.gpg
RUN echo 'deb [signed-by=/usr/share/keyrings/nvidia-hpcsdk-archive-keyring.gpg] https://developer.download.nvidia.com/hpc-sdk/ubuntu/amd64 /' |  tee /etc/apt/sources.list.d/nvhpc.list
RUN apt-get update -y
RUN apt-get install -y nvhpc-24-3
RUN apt install -f

#TENSORRT
RUN wget https://developer.nvidia.com/downloads/compute/machine-learning/tensorrt/secure/8.6.1/local_repos/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0_1.0-1_amd64.deb
RUN dpkg -i nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0_1.0-1_amd64.deb
RUN cp /var/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0/*-keyring.gpg /usr/share/keyrings/
RUN apt-get update
RUN apt-get install -y tensorrt
RUN apt-get install -y libnvinfer-lean8
RUN apt-get install -y libnvinfer-vc-plugin8
RUN python3 -m pip install numpy

RUN dpkg-query -W tensorrt
#tensorrt	10.0.0.6-1+cuda12.4

RUN conda install python=3.9.18

RUN pip install boto3 spleeter tensorrt cuda-python
RUN pip install git+https://github.com/facebookresearch/audiocraft.git
RUN pip install torch==2.1.0+cu121 --index-url https://download.pytorch.org/whl/cu121

RUN conda install -c nvidia cuda-python cudatoolkit

RUN export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/cuda-12.0/targets/x86_64-linux/lib

RUN find / -name '*libcudart*'
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/REDIST/cuda/12.3/targets/x86_64-linux/lib/libcudart.so.12.3.101
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/REDIST/cuda/12.3/targets/x86_64-linux/lib/libcudart.so.12
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/REDIST/cuda/12.3/targets/x86_64-linux/lib/libcudart.so
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/cuda/12.3/targets/x86_64-linux/lib/libcudart.so.12
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/cuda/12.3/targets/x86_64-linux/lib/libcudart.so
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/cuda/12.3/targets/x86_64-linux/lib/libcudart.so.12.3.101
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/cuda/12.3/targets/x86_64-linux/lib/libcudart_static.a
/root/miniconda3/lib/python3.9/site-packages/nvidia/cuda_runtime/lib/libcudart.so.12
/root/miniconda3/lib/python3.9/site-packages/torch/lib/libcudart-9335f6a2.so.12
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudart.so.12
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudart.so.12.0.107
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudart.so
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudart_static.a

RUN find / -name '*tensorrt*'
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/REDIST/cuda/12.3/targets/x86_64-linux/lib/libcudart.so.12.3.101
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/REDIST/cuda/12.3/targets/x86_64-linux/lib/libcudart.so.12
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/REDIST/cuda/12.3/targets/x86_64-linux/lib/libcudart.so
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/cuda/12.3/targets/x86_64-linux/lib/libcudart.so.12
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/cuda/12.3/targets/x86_64-linux/lib/libcudart.so
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/cuda/12.3/targets/x86_64-linux/lib/libcudart.so.12.3.101
/opt/nvidia/hpc_sdk/Linux_x86_64/24.3/cuda/12.3/targets/x86_64-linux/lib/libcudart_static.a
/root/miniconda3/lib/python3.9/site-packages/nvidia/cuda_runtime/lib/libcudart.so.12
/root/miniconda3/lib/python3.9/site-packages/torch/lib/libcudart-9335f6a2.so.12
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudart.so.12
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudart.so.12.0.107
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudart.so
/usr/local/cuda-12.0/targets/x86_64-linux/lib/libcudart_static.a
Expand

=> Step 35: RUN find / -name '*tensorrt*'
/root/miniconda3/lib/python3.9/site-packages/tensorrt_bindings
/root/miniconda3/lib/python3.9/site-packages/tensorrt_bindings/tensorrt.so
/root/miniconda3/lib/python3.9/site-packages/tensorrt_bindings-8.6.1.dist-info
/root/miniconda3/lib/python3.9/site-packages/tensorrt_libs
/root/miniconda3/lib/python3.9/site-packages/tensorrt_libs-8.6.1.dist-info
/root/miniconda3/lib/python3.9/site-packages/tensorrt
/root/miniconda3/lib/python3.9/site-packages/tensorrt-8.6.1.post1.dist-info
/root/miniconda3/lib/python3.9/site-packages/tensorflow/python/compiler/tensorrt
/root/miniconda3/lib/python3.9/site-packages/tensorflow/_api/v2/compat/v2/experimental/tensorrt
/root/miniconda3/lib/python3.9/site-packages/tensorflow/_api/v2/experimental/tensorrt
/root/miniconda3/lib/python3.9/site-packages/tensorflow/compiler/tf2tensorrt
/root/miniconda3/lib/python3.9/site-packages/tensorflow/include/tensorflow/compiler/tf2tensorrt
/root/miniconda3/lib/python3.9/site-packages/torch/_dynamo/backends/tensorrt.py
/root/miniconda3/lib/python3.9/site-packages/torch/_dynamo/backends/__pycache__/tensorrt.cpython-39.pyc
/root/miniconda3/lib/python3.9/site-packages/torch/ao/quantization/backend_config/tensorrt.py
/root/miniconda3/lib/python3.9/site-packages/torch/ao/quantization/backend_config/__pycache__/tensorrt.cpython-39.pyc
/usr/share/doc/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0
/usr/share/doc/tensorrt-10.0.0.6
/usr/share/doc/tensorrt
/usr/share/keyrings/nv-tensorrt-local-9A1EDFBA-keyring.gpg
/usr/lib/python3.8/dist-packages/tensorrt
/usr/lib/python3.8/dist-packages/tensorrt/tensorrt.so
/usr/lib/python3.8/dist-packages/tensorrt-10.0.0b6.dist-info
/usr/lib/python3.8/dist-packages/tensorrt_lean
/usr/lib/python3.8/dist-packages/tensorrt_lean/tensorrt_lean.so
/usr/lib/python3.8/dist-packages/tensorrt_lean-10.0.0b6.dist-info
/usr/lib/python3.8/dist-packages/tensorrt_dispatch
/usr/lib/python3.8/dist-packages/tensorrt_dispatch/tensorrt_dispatch.so
/usr/lib/python3.8/dist-packages/tensorrt_dispatch-10.0.0b6.dist-info
/usr/src/tensorrt
/usr/src/tensorrt/samples/python/yolov3_onnx/onnx_to_tensorrt.py
/etc/apt/sources.list.d/nv-tensorrt-local-ubuntu2004-8.6.1-cuda-12.0.list
/var/lib/apt/lists/_var_nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0_InRelease
/var/lib/apt/lists/_var_nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0_Packages.lz4
/var/lib/dpkg/info/tensorrt.md5sums
/var/lib/dpkg/info/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0.list
/var/lib/dpkg/info/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0.conffiles
/var/lib/dpkg/info/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0.md5sums
/var/lib/dpkg/info/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0.postinst
/var/lib/dpkg/info/tensorrt.list
/var/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0
/var/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0/nv-tensorrt-local-9A1EDFBA-keyring.gpg
/var/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0/tensorrt-dev_8.6.1.6-1+cuda12.0_amd64.deb
/var/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0/tensorrt-libs_8.6.1.6-1+cuda12.0_amd64.deb
/var/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0/tensorrt_8.6.1.6-1+cuda12.0_amd64.deb
/nv-tensorrt-local-repo-ubuntu2004-8.6.1-cuda-12.0_1.0-1_amd64.deb


RUN python3 -c "import tensorrt; print(tensorrt.__path__)"
RUN python3 -c "import tensorflow as tf; print(tf.config.list_physical_devices('GPU'))"
RUN python -c "import tensorflow.compiler as tf_cc; print(tf_cc.tf2tensorrt._pywrap_py_utils.get_linked_tensorrt_version())"
RUN pip list

RUN nvcc --version
#nvcc: NVIDIA (R) Cuda compiler driver
#Copyright (c) 2005-2023 NVIDIA Corporation
#Built on Fri_Jan__6_16:45:21_PST_2023
#Cuda compilation tools, release 12.0, V12.0.140
#Build cuda_12.0.r12.0/compiler.32267302_0

RUN lsb_release -a
# No LSB modules are available.
# Distributor ID: Ubuntu
# Description:    Ubuntu 20.04.5 LTS
# Release:        20.04
# Codename:       focal

google-ml-butler bot added the type:others issues not falling in bug, perfromance, support, build and install or feature label Feb 16, 2024

google-ml-butler bot assigned sushreebarsa Feb 16, 2024

tymorrow changed the title ~~TF_USE_TENSORRT_STATIC~~ Understanding Could not find TensorRT Feb 16, 2024

sushreebarsa added comp:gpu:tensorrt Issues specific to TensorRT TF 2.15 For issues related to 2.15.x labels Feb 19, 2024

sushreebarsa added the stat:awaiting response Status - Awaiting response from author label Feb 19, 2024

google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label Feb 21, 2024

sushreebarsa assigned sachinprasadhs and unassigned sushreebarsa Feb 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Understanding `Could not find TensorRT` #62976

Understanding `Could not find TensorRT` #62976

Understanding Could not find TensorRT #62976

Understanding Could not find TensorRT #62976

Comments

Issue type

Have you reproduced the bug with TensorFlow Nightly?

Source

TensorFlow version

Custom code

OS platform and distribution

Mobile device

Python version

Bazel version

GCC/compiler version

CUDA/cuDNN version

GPU model and memory

Current behavior?

Standalone code to reproduce the issue

Relevant log output

Understanding `Could not find TensorRT` #62976

Understanding `Could not find TensorRT` #62976