Failure to compile TF 2.15.0/2.16.1 (with Cuda support) using clang in Ubuntu 22.04 #62459

feranick · 2023-11-22T18:58:51Z

Issue type

Build/Install

Have you reproduced the bug with TensorFlow Nightly?

No

Source

source

TensorFlow version

2.15.0/2.16.1

Custom code

No

OS platform and distribution

Linux Ubuntu 16.04

Mobile device

No response

Python version

3.10.12

Bazel version

6.1.0

GCC/compiler version

Clang 14.0.0-1ubuntu1.1

CUDA/cuDNN version

11.8-12.3

GPU model and memory

Quadro RTX 6000 24GB

Current behavior?

Compiling TF 2.15.0/2.16.1 in Ubuntu 22.04 using clang fails. Log is attached.

Standalone code to reproduce the issue

Using this configuration: 

./configure
You have bazel 6.1.0 installed. (6.5 for TF 2.16.1)
Please specify the location of python. [Default is /usr/bin/python3]: 


Found possible Python library paths:
  /usr/lib/python3/dist-packages
  /usr/local/lib/python3.10/dist-packages
Please input the desired Python library path to use.  Default is [/usr/lib/python3/dist-packages]

Do you wish to build TensorFlow with ROCm support? [y/N]: 
No ROCm support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: y
CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with TensorRT support? [y/N]: y
TensorRT support will be enabled for TensorFlow.

Found CUDA 11.8 in:
    /usr/local/cuda-11.8/targets/x86_64-linux/lib
    /usr/local/cuda-11.8/targets/x86_64-linux/include
Found cuDNN 8 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include
Found TensorRT 8.5.3 in:
    /usr/lib/x86_64-linux-gnu
    /usr/include/x86_64-linux-gnu



Please specify a list of comma-separated CUDA compute capabilities you want to build with.
You can find the compute capability of your device at: https://developer.nvidia.com/cuda-gpus. Each capability can be specified as "x.y" or "compute_xy" to include both virtual and binary GPU code, or as "sm_xy" to only include the binary code.
Please note that each additional compute capability significantly increases your build time and binary size, and that TensorFlow only supports compute capabilities >= 3.5 [Default is: 7.5,7.5]: 

Do you want to use clang as CUDA compiler? [Y/n]: 
Clang will be used as CUDA compiler.

Please specify clang path that to be used as host compiler. [Default is /usr/bin/clang]: 


You have Clang 14.0.0-1ubuntu1.1 installed.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -Wno-sign-compare]: -Wno-sign-compare


Would you like to interactively configure ./WORKSPACE for Android builds? [y/N]: 
Not configuring the WORKSPACE for Android builds.

Preconfigured Bazel build configs. You can use any of the below by adding "--config=<>" to your build command. See .bazelrc for more details.
	--config=mkl         	# Build with MKL support.
	--config=mkl_aarch64 	# Build with oneDNN and Compute Library for the Arm Architecture (ACL).
	--config=monolithic  	# Config for mostly static monolithic build.
	--config=numa        	# Build with NUMA support.
	--config=dynamic_kernels	# (Experimental) Build kernels into separate shared objects.
	--config=v1          	# Build with TensorFlow 1 API instead of TF 2 API.
Preconfigured Bazel build configs to DISABLE default on features:
	--config=nogcp       	# Disable GCP support.
	--config=nonccl      	# Disable NVIDIA NCCL support.
Configuration finished

Relevant log output

Extracting Bazel installation...
Starting local Bazel server and connecting to it...
WARNING: The following configs were expanded more than once: [tensorrt, cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
INFO: Reading 'startup' options from /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --windows_enable_symlinks
INFO: Options provided by the client:
  Inherited 'common' options: --isatty=1 --terminal_columns=100
INFO: Reading rc options for 'build' from /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc:
  Inherited 'common' options: --experimental_repo_remote_exec
INFO: Reading rc options for 'build' from /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc:
  'build' options: --define framework_shared_object=true --define tsl_protobuf_header_only=true --define=use_fast_cpp_protos=true --define=allow_oversize_protos=true --spawn_strategy=standalone -c opt --announce_rc --define=grpc_no_ares=true --noincompatible_remove_legacy_whole_archive --features=-force_no_whole_archive --enable_platform_specific_config --define=with_xla_support=true --config=short_logs --config=v2 --define=no_aws_support=true --define=no_hdfs_support=true --experimental_cc_shared_library --experimental_link_static_libraries_once=false --incompatible_enforce_config_setting_visibility
INFO: Reading rc options for 'build' from /home/nicola/Software/tensorflow/gpu/tensorflow/.tf_configure.bazelrc:
  'build' options: --action_env PYTHON_BIN_PATH=/usr/bin/python3 --action_env PYTHON_LIB_PATH=/usr/lib/python3/dist-packages --python_path=/usr/bin/python3 --config=tensorrt --action_env CUDA_TOOLKIT_PATH=/usr/local/cuda-11.8 --action_env TF_CUDA_COMPUTE_CAPABILITIES=7.5,7.5 --action_env LD_LIBRARY_PATH=/usr/local/cuda-11.8/include:/usr/local/cuda-11.8/targets/x86_64-linux/lib: --config=cuda_clang --action_env CLANG_CUDA_COMPILER_PATH=/usr/lib/llvm-14/bin/clang --config=cuda_clang
INFO: Found applicable config definition build:short_logs in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --output_filter=DONT_MATCH_ANYTHING
INFO: Found applicable config definition build:v2 in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --define=tf_api_version=2 --action_env=TF2_BEHAVIOR=1
INFO: Found applicable config definition build:tensorrt in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_TENSORRT=1
INFO: Found applicable config definition build:cuda_clang in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --config=cuda --config=tensorrt --action_env=TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang --repo_env=TF_CUDA_COMPUTE_CAPABILITIES=sm_50,sm_60,sm_70,sm_75,compute_80
INFO: Found applicable config definition build:cuda in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:tensorrt in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_TENSORRT=1
INFO: Found applicable config definition build:cuda_clang in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --config=cuda --config=tensorrt --action_env=TF_CUDA_CLANG=1 --@local_config_cuda//:cuda_compiler=clang --repo_env=TF_CUDA_COMPUTE_CAPABILITIES=sm_50,sm_60,sm_70,sm_75,compute_80
INFO: Found applicable config definition build:cuda in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:tensorrt in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_TENSORRT=1
INFO: Found applicable config definition build:opt in file /home/nicola/Software/tensorflow/gpu/tensorflow/.tf_configure.bazelrc: --copt=-Wno-sign-compare --host_copt=-Wno-sign-compare
INFO: Found applicable config definition build:cuda in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --repo_env TF_NEED_CUDA=1 --crosstool_top=@local_config_cuda//crosstool:toolchain --@local_config_cuda//:enable_cuda
INFO: Found applicable config definition build:linux in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --host_copt=-w --copt=-Wno-all --copt=-Wno-extra --copt=-Wno-deprecated --copt=-Wno-deprecated-declarations --copt=-Wno-ignored-attributes --copt=-Wno-array-bounds --copt=-Wunused-result --copt=-Werror=unused-result --copt=-Wswitch --copt=-Werror=switch --copt=-Wno-error=unused-but-set-variable --define=PREFIX=/usr --define=LIBDIR=$(PREFIX)/lib --define=INCLUDEDIR=$(PREFIX)/include --define=PROTOBUF_INCLUDE_PATH=$(PREFIX)/include --cxxopt=-std=c++17 --host_cxxopt=-std=c++17 --config=dynamic_kernels --experimental_guard_against_concurrent_changes
INFO: Found applicable config definition build:dynamic_kernels in file /home/nicola/Software/tensorflow/gpu/tensorflow/.bazelrc: --define=dynamic_loaded_kernels=true --copt=-DAUTOLOAD_DYNAMIC_KERNELS
DEBUG: /home/nicola/Software/tensorflow/gpu/tensorflow/tensorflow/tools/toolchains/python/python_repo.bzl:21:14: 
TF_PYTHON_VERSION variable was not set correctly, using default version. 3.10 Python
will be used.

To set Python version, run
export TF_PYTHON_VERSION=3.9
WARNING: The following configs were expanded more than once: [tensorrt, cuda_clang, cuda]. For repeatable flags, repeats are counted twice and may lead to unexpected behavior.
WARNING: Download from https://mirror.bazel.build/github.com/bazelbuild/rules_cc/archive/081771d4a0e9d7d3aa0eed2ef389fa4700dfb23e.tar.gz failed: class java.io.FileNotFoundException GET returned 404 Not Found
INFO: Analyzed target //tensorflow/tools/pip_package:build_pip_package (699 packages loaded, 49725 targets configured).
INFO: Found 1 target...
ERROR: /home/nicola/.cache/bazel/_bazel_nicola/c53ed0be17816f9e0970b1ba234e403c/external/com_google_protobuf/BUILD.bazel:27:11: Compiling src/google/protobuf/arenaz_sampler.cc [for tool] failed: (Exit 1): clang failed: error executing command (from target @com_google_protobuf//:protobuf_lite) 
  (cd /home/nicola/.cache/bazel/_bazel_nicola/c53ed0be17816f9e0970b1ba234e403c/execroot/org_tensorflow && \
  exec env - \
    LD_LIBRARY_PATH=/usr/local/cuda-11.8/include:/usr/local/cuda-11.8/targets/x86_64-linux/lib: \
    PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/snap/bin \
    PWD=/proc/self/cwd \
  /usr/lib/llvm-14/bin/clang -MD -MF bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/_objs/protobuf_lite/arenaz_sampler.d '-frandom-seed=bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/_objs/protobuf_lite/arenaz_sampler.o' '-DBAZEL_CURRENT_REPOSITORY="com_google_protobuf"' -iquote external/com_google_protobuf -iquote bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf -isystem external/com_google_protobuf/src -isystem bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/src -fmerge-all-constants -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIE -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -Wno-invalid-partial-specialization -fno-omit-frame-pointer -no-canonical-prefixes -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections '--cuda-path=/usr/local/cuda-11.8' -g0 -w -Wno-sign-compare -g0 '-std=c++17' -DHAVE_ZLIB -Woverloaded-virtual -Wno-sign-compare -c external/com_google_protobuf/src/google/protobuf/arenaz_sampler.cc -o bazel-out/k8-opt-exec-50AE0418/bin/external/com_google_protobuf/_objs/protobuf_lite/arenaz_sampler.o)
# Configuration: fccf36a8de6040562b99997447208dc7348eb48df9f6a445d4378df7e13a876c
# Execution platform: @local_execution_config_platform//:platform
In file included from external/com_google_protobuf/src/google/protobuf/arenaz_sampler.cc:31:
external/com_google_protobuf/src/google/protobuf/arenaz_sampler.h:34:10: fatal error: 'atomic' file not found
#include <atomic>
         ^~~~~~~~
1 error generated.
Target //tensorflow/tools/pip_package:build_pip_package failed to build
INFO: Elapsed time: 84.960s, Critical Path: 0.43s
INFO: 60 processes: 59 internal, 1 local.
FAILED: Build did NOT complete successfully

feranick · 2023-11-22T19:28:51Z

When nvcc is used as compiler, compilation still fails. Configure and compile Logs are attached.
nvcc_compile.txt
nvcc_config.txt

feranick · 2023-11-22T19:29:23Z

The only way to have a successful compilation is to use gcc with NO Cuda support.

feranick · 2023-11-22T22:06:47Z

The original bug while using clang can be fixed by making sure the library libstdc++-12-dev is installed:

sudo apt install libstdc++-12-dev

However a new set of errors appear, which seem related to a confusing definition of noinline. Log:

ERROR: /home/nicola/Software/tensorflow-dir/gpu/tensorflow/tensorflow/core/kernels/BUILD:5142:18: Compiling tensorflow/core/kernels/sparse_to_dense_op_gpu.cu.cc failed: (Exit 1): clang failed: error executing command (from target //tensorflow/core/kernels:sparse_to_dense_op_gpu) /usr/lib/llvm-14/bin/clang -MD -MF bazel-out/k8-opt/bin/tensorflow/core/kernels/_objs/sparse_to_dense_op_gpu/sparse_to_dense_op_gpu.cu.pic.d ... (remaining 221 arguments skipped)
clang: warning: CUDA version is newer than the latest supported version 11.5 [-Wunknown-cuda-version]
In file included from tensorflow/core/kernels/sparse_to_dense_op_gpu.cu.cc:19:
In file included from ./tensorflow/core/kernels/sparse_to_dense_op_gpu.h:23:
In file included from external/local_xla/xla/stream_executor/device_memory.h:29:
In file included from external/local_xla/xla/stream_executor/platform/port.h:22:
In file included from external/local_tsl/tsl/platform/types.h:24:
In file included from external/local_tsl/tsl/platform/tstring.h:24:
In file included from external/local_tsl/tsl/platform/cord.h:21:
In file included from external/com_google_absl/absl/strings/cord.h:74:
In file included from external/com_google_absl/absl/base/internal/endian.h:22:
In file included from external/com_google_absl/absl/base/casts.h:28:
In file included from /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/memory:76:
In file included from /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr.h:53:
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr_base.h:196:22: error: use of undeclared identifier 'noinline'; did you mean 'inline'?
      __attribute__((__noinline__))
                     ^
bazel-out/k8-opt/bin/external/local_config_cuda/cuda/cuda/include/crt/host_defines.h:83:24: note: expanded from macro '__noinline__'
        __attribute__((noinline))
                       ^
In file included from tensorflow/core/kernels/sparse_to_dense_op_gpu.cu.cc:19:
In file included from ./tensorflow/core/kernels/sparse_to_dense_op_gpu.h:23:
In file included from external/local_xla/xla/stream_executor/device_memory.h:29:
In file included from external/local_xla/xla/stream_executor/platform/port.h:22:
In file included from external/local_tsl/tsl/platform/types.h:24:
In file included from external/local_tsl/tsl/platform/tstring.h:24:
In file included from external/local_tsl/tsl/platform/cord.h:21:
In file included from external/com_google_absl/absl/strings/cord.h:74:
In file included from external/com_google_absl/absl/base/internal/endian.h:22:
In file included from external/com_google_absl/absl/base/casts.h:28:
In file included from /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/memory:76:
In file included from /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr.h:53:
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr_base.h:196:22: error: type name does not allow function specifier to be specified
bazel-out/k8-opt/bin/external/local_config_cuda/cuda/cuda/include/crt/host_defines.h:83:24: note: expanded from macro '__noinline__'
        __attribute__((noinline))
                       ^
In file included from tensorflow/core/kernels/sparse_to_dense_op_gpu.cu.cc:19:
In file included from ./tensorflow/core/kernels/sparse_to_dense_op_gpu.h:23:
In file included from external/local_xla/xla/stream_executor/device_memory.h:29:
In file included from external/local_xla/xla/stream_executor/platform/port.h:22:
In file included from external/local_tsl/tsl/platform/types.h:24:
In file included from external/local_tsl/tsl/platform/tstring.h:24:
In file included from external/local_tsl/tsl/platform/cord.h:21:
In file included from external/com_google_absl/absl/strings/cord.h:74:
In file included from external/com_google_absl/absl/base/internal/endian.h:22:
In file included from external/com_google_absl/absl/base/casts.h:28:
In file included from /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/memory:76:
In file included from /usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr.h:53:
/usr/lib/gcc/x86_64-linux-gnu/12/../../../../include/c++/12/bits/shared_ptr_base.h:196:22: error: expected expression
bazel-out/k8-opt/bin/external/local_config_cuda/cuda/cuda/include/crt/host_defines.h:83:33: note: expanded from macro '__noinline__'
        __attribute__((noinline))
                                ^

sushreebarsa · 2023-11-27T09:43:44Z

@feranick Could you please try to uninstall the clang version 14 and install the clang 13 and try to reconfigure TF ? Please let us know if it helps?
Thank you!

feranick · 2023-11-27T20:21:42Z

Compilation still fails, although for a different reason. Log attached.
clang-13.txt

feranick · 2023-11-27T21:11:06Z

Actually, I tried it in a different machine with a similar configuration, and even using clang-13 the error seems to be the same as in the original post (using clang-14). Log attached.

clang-13_2.txt

feranick · 2023-11-27T23:44:34Z

As the issue seems to be related to CUDA, what is the preferred version of CUDA that Google advises to use for TF2.15.0? TF2.14.x works with CUDA 11.8, but this doesn't work (as per this bug) for TF2.15.0.

sachinprasadhs · 2023-11-28T06:17:04Z

As the issue seems to be related to CUDA, what is the preferred version of CUDA that Google advises to use for TF2.15.0? TF2.14.x works with CUDA 11.8, but this doesn't work (as per this bug) for TF2.15.0.

12.2 is the CUDA version we bumped from TensorFlow 2.15. Could you please try with that and let us know the outcome. Thanks!

feranick · 2023-11-28T15:35:33Z

Still getting the same error with CUDA 12.2. Log attached.
clang_3.txt

feranick · 2023-11-28T18:07:13Z

I just tried to compile TF2.15.0 using CUDA 12.2 and nvcc (NOT clang) and compilation was successful. I'd leave this bug report open, as compilation with clang still fails (and clang is now the default compiler in configure).

feranick · 2023-11-28T19:49:04Z

It would also be good to update the ./configure to reflect that the min version of CUDA should be 12.2 (it still lists 11), and somewhere a list of the supported versions of each software required.

sachinprasadhs · 2023-11-29T22:48:59Z

We have updated the document in the code, due to some sync issue it is not published yet in the website.

nadyawilliams · 2023-12-06T22:42:45Z

I have the same problem on rocky 8.8 compiling tensorflow 2.15 with cuda 12.2 and clang 16.0.1.

Modules loaded for compilation (all compiled in house on the same host):

bazel/6.1.0  cuda/12.2.0  python/3.10.2  tensorRT/8.6.1.6  java/11  gcc/11.2.0  llvm/16.0.1  clang/16.0.1

Running .configure results in the following .tf_configure.bazelrc (implying using clang as cuda compiler):

build --action_env PYTHON_BIN_PATH="/opt/apps/python/3.10.2/bin/python3"
build --action_env PYTHON_LIB_PATH="/opt/apps/python/3.10.2/lib/python3.10/site-packages"
build --python_path="/opt/apps/python/3.10.2/bin/python3"
build --config=tensorrt
build --action_env TF_CUDA_VERSION="12"
build --action_env TF_CUDNN_VERSION="8"
build --action_env TF_TENSORRT_VERSION="8"
build --action_env TF_NCCL_VERSION=""
build --action_env TF_CUDA_PATHS="/opt/apps/cuda/12.2.0,/opt/apps/tensorRT/8.6.1.6,/usr"
build --action_env CUDA_TOOLKIT_PATH="/opt/apps/cuda/12.2.0"
build --action_env TF_CUDA_COMPUTE_CAPABILITIES="7.0,8.0"
build --action_env LD_LIBRARY_PATH="/opt/apps/clang/16.0.1/lib:/opt/apps/llvm/16.0.1/lib:/opt/apps/gcc/11.2.0/lib64:/opt/apps/gcc/11.2.0/lib:/opt/apps/gcc/11.2.0/lib/gcc/x86_64-pc-linux-gnu/11.2.0:/opt/apps/tensorRT/8.6.1.6/lib:/opt/apps/python/3.10.2
/lib:/opt/apps/cuda/12.2.0/lib64:/opt/apps/cuda/12.2.0/nvvm/lib64:/opt/apps/cuda/12.2.0/cublas/lib64:/opt/apps/cuda/12.2.0/extras/CUPTI/lib64:/opt/apps/cuda/12.2.0/extras/Debugger/lib64"
build --config=cuda_clang
build --action_env CLANG_CUDA_COMPILER_PATH="/opt/apps/clang/16.0.1/bin/clang-16"
build --copt=-Wno-gnu-offsetof-extensions
build --config=cuda_clang
build:opt --copt=-mavx2
build:opt --host_copt=-mavx2
test --test_size_filters=small,medium
test --test_env=LD_LIBRARY_PATH
test:v1 --test_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-gpu,-oss_serial
test:v1 --build_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-gpu
test:v2 --test_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-gpu,-oss_serial,-v1only
test:v2 --build_tag_filters=-benchmark-test,-no_oss,-oss_excluded,-gpu,-v1only

The bazel build command:

bazel build --config=opt --jobs=8 --verbose_failures --verbose_explanations --explain=/tmp/explain \
                      //tensorflow/tools/pip_package:build_pip_package

If i rerun configure and choose not to use clang as cuda compiler it results in a slightly
different .tf.configure.bazelrc file for brevity here are the diffs:

dont use clang
< build --action_env GCC_HOST_COMPILER_PATH="/opt/apps/gcc/11.2.0/bin/gcc"
< build --config=cuda
---
use clang
> build --config=cuda_clang
> build --action_env CLANG_CUDA_COMPILER_PATH="/opt/apps/clang/16.0.1/bin/clang-16"
> build --copt=-Wno-gnu-offsetof-extensions
> build --config=cuda_clang

and the bazel build fails with a different error:

bazel-out/k8-opt-exec-50AE0418/bin/tensorflow/python/framework/offset_counter: /lib64/libstdc++.so.6: version 'CXXABI_1.3.13' not found (required by /root/.cache/bazel/_bazel_root/e76370378c3e9e8e238b869c10fc760e/execroot/org_tensorflow/bazel-out/k8-opt-exec-50AE0418/bin/tensorflow/python/framework/../../../_solib_local/_U_S_Stensorflow_Spython_Sframework_Coffset_Ucounter___Utensorflow/libtensorflow_framework.so.2)
bazel-out/k8-opt-exec-50AE0418/bin/tensorflow/python/framework/offset_counter: /lib64/libstdc++.so.6: version 'GLIBCXX_3.4.29' not found (required by /root/.cache/bazel/_bazel_root/e76370378c3e9e8e238b869c10fc760e/execroot/org_tensorflow/bazel-out/k8-opt-exec-50AE0418/bin/tensorflow/python/framework/../../../_solib_local/_U_S_Stensorflow_Spython_Sframework_Coffset_Ucounter___Utensorflow/libtensorflow_framework.so.2)
bazel-out/k8-opt-exec-50AE0418/bin/tensorflow/python/framework/offset_counter: /lib64/libstdc++.so.6: version 'GLIBCXX_3.4.26' not found (required by /root/.cache/bazel/_bazel_root/e76370378c3e9e8e238b869c10fc760e/execroot/org_tensorflow/bazel-out/k8-opt-exec-50AE0418/bin/tensorflow/python/framework/../../../_solib_local/_U_S_Stensorflow_Spython_Sframework_Coffset_Ucounter___Utensorflow/libtensorflow_framework.so.2)

While the compilation itself apparently went further, it makes no sense to me that bazel subcommand failed with
above errors because /lib64/libstdc++.so.6 should not be used as my LD_LIBRARY_PATH is set to use clang 16
and gcc 11.2.0 path first and gcc compilation includes libstdc++.so.6 that has all above mentioned CXX and GLIBCXX entries. Bazel binary and clang libraries are linked to use that correct libstdc++.so.6 from gcc 11.
The question is where does the /lib64/libstdc++.so.6 come from in bazel work?

Either way, with or without clang bazel build fails.

feranick · 2023-12-06T22:51:17Z

This is a separate issues than the one reported here (which is specific to Ubuntu, and related to the way CUDA is called within clang). I would recommend filing a separate issue for Rocky linux and libstfc++.

targettadams · 2023-12-11T20:58:29Z

I have also been trying to compile TF 2.15 (well, master actually) on Ubuntu 22.04 and have been running into some issues with clang.

I have to say, I find the available documentation on compiling TF from source a tad contradictory and a tad confusing. My story so far is as follows.

Install cuda 12.2 and clang 17 (as per 2.15 release notes).
Install cuDnn 8.9.7.

I then tried to compile master, but it fails because, despite not configuring tensorrt, it is looking for tensorrt headers.

So I try to install tensorrt, but it does not support cuda 12.2 yet.
Install cuda 12.1
Install tensorrt
Install libstdc++-12-dev

Compiling with clang gets me deep into the compilation, but it then bombs out:

ERROR: /home/christopher/.cache/bazel/_bazel_christopher/696f0d61a4ca964b6d054bb99f460f18/external/upb/BUILD:57:11: Compiling upb/decode.c failed: (Exit 1): clang failed: error executing command (from target @upb//:upb) /usr/lib/llvm-17/bin/clang -MD -MF bazel-out/k8-opt/bin/external/upb/_objs/upb/decode.pic.d '-frandom-seed=bazel-out/k8-opt/bin/external/upb/_objs/upb/decode.pic.o' '-DBAZEL_CURRENT_REPOSITORY="upb"' ... (remaining 43 arguments skipped)
clang: error: argument unused during compilation: '--cuda-path=/usr/local/cuda-12.1' [-Werror,-Wunused-command-line-argument]
Target //tensorflow/tools/pip_package:build_pip_package failed to build

I am now attempting the build with gcc ...

feranick · 2024-03-12T15:59:34Z

Still an issue with TF 2.1.6.1. Apparently compiling with clang supports only CUDA up to v11.5. See log below.
log.txt

sachinprasadhs · 2024-03-12T22:19:56Z

Have you tried it with Clang 17 on your ubuntu, here is the doc for installing Clang https://www.tensorflow.org/install/source#install_clang_recommended_linux_only

feranick · 2024-03-13T13:37:37Z

I will. However, the updated clang 17 is not available in the standard Ubuntu repository (it's an external repo), which means that TF cannot be compiled wth Clang with standard tools. gcc works fine. It would be great to mention that during ./configure (at least the min version of clang required.

QuesarVII · 2024-03-21T02:37:08Z

I was able to build 2.16.1 with CUDA 12.4 with clang 17 on Ubuntu 22.04 by manually adding the line "build:cuda --copt=-Wno-error=unused-command-line-argument" into .tf_configure.bazelrc after running configure and before running bazel build. I also had to "export TF_PYTHON_VERSION=3.10" before "bazel build --subcommands //tensorflow/tools/pip_package/v2:wheel --repo_env=WHEEL_NAME=tensorflow --config=cuda". I do have a separate build error (see issue 62047) if I add copt "-march=native" or "-mavx" though.

QuesarVII · 2024-03-21T18:31:40Z

Minor correction - the issue referenced in 62047 only happens with "-march=native". "-mavx" does build and run successfully. The other issue was blocked from comments since I added my note there. This was done with Intel 4410Y CPUs.

mahmoudimus · 2024-04-01T18:21:50Z

I was able to build 2.16.1 with CUDA 12.4 with clang 17 on Ubuntu 22.04 by manually adding the line "build:cuda --copt=-Wno-error=unused-command-line-argument" into .tf_configure.bazelrc after running configure and before running bazel build. I also had to "export TF_PYTHON_VERSION=3.10" before "bazel build --subcommands //tensorflow/tools/pip_package/v2:wheel --repo_env=WHEEL_NAME=tensorflow --config=cuda". I do have a separate build error (see issue 62047) if I add copt "-march=native" or "-mavx" though.

You don't have to modify the .tf_configure.bazelrc file, but yes, if compiling with clang-17, then you should pass --copt=-Wno-error=unused-command-line-argument to the bazel build command and it will accomplish the same goal w/o modifying a file etc.

shaofan-qi · 2024-04-09T00:47:04Z

I built v2.16.1 successfully with CUDA 12.3 + cuDNN 8.9.7 + tensorrt 8.6.1 with clang 17 on Ubuntu 22.04, with command line option --copt=-Wno-error=unused-command-line-argument.

plopresti · 2024-06-26T15:57:21Z

Still present on r2.17. Any chance of seeing this fixed?

google-ml-butler bot added the type:build/install Build and install issues label Nov 22, 2023

google-ml-butler bot assigned tilakrayal Nov 22, 2023

sushreebarsa assigned sushreebarsa and unassigned tilakrayal Nov 23, 2023

sushreebarsa added TF 2.15 For issues related to 2.15.x subtype: ubuntu/linux Ubuntu/Linux Build/Installation Issues labels Nov 27, 2023

sushreebarsa added the stat:awaiting response Status - Awaiting response from author label Nov 27, 2023

google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label Nov 27, 2023

sushreebarsa assigned sachinprasadhs and unassigned sushreebarsa Nov 28, 2023

sachinprasadhs added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Nov 29, 2023

sachinprasadhs mentioned this issue Nov 29, 2023

Install error with clang compiler. CUDA device code does not support variadict functions. #62339

Open

feranick changed the title ~~Failure to compile TF 2.15.0 (with Cuda support) using clang in Ubuntu 22.04~~ Failure to compile TF 2.15.0/2.16.1 (with Cuda support) using clang in Ubuntu 22.04 Mar 12, 2024

sachinprasadhs added stat:awaiting response Status - Awaiting response from author and removed stat:awaiting tensorflower Status - Awaiting response from tensorflower labels Mar 12, 2024

google-ml-butler bot removed the stat:awaiting response Status - Awaiting response from author label Mar 13, 2024

sachinprasadhs added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Mar 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Failure to compile TF 2.15.0/2.16.1 (with Cuda support) using clang in Ubuntu 22.04 #62459

Failure to compile TF 2.15.0/2.16.1 (with Cuda support) using clang in Ubuntu 22.04 #62459

Failure to compile TF 2.15.0/2.16.1 (with Cuda support) using clang in Ubuntu 22.04 #62459

Failure to compile TF 2.15.0/2.16.1 (with Cuda support) using clang in Ubuntu 22.04 #62459

Comments

Issue type

Have you reproduced the bug with TensorFlow Nightly?

Source

TensorFlow version

Custom code

OS platform and distribution

Mobile device

Python version

Bazel version

GCC/compiler version

CUDA/cuDNN version

GPU model and memory

Current behavior?

Standalone code to reproduce the issue

Relevant log output