I have the same problems, I have checked the driver(ok), cuda(ok), cudnn(ok), nccl(ok), all of these were normal operation. However, the distribution training was broken and frozen. It is so disappointing.

Do you have any logs that I can look into?

Commenting just to say that I'm seeing the same behavior though with cuDNN version 8400

EDIT: Felt this might be useful - If I use

strategy = tf.distribute.MirroredStrategy(devices=['/gpu:0'])

things work fine. If I switch it to /gpu:1, I get the following errors:

...
Node: 'Assert/Assert'
3 root error(s) found.
  (0) INVALID_ARGUMENT:  assertion failed: [loop must iterate at least once to initialize outputs]
         [[{{node Assert/Assert}}]]
         [[while/body/_1/while/div_no_nan_1/ReadVariableOp/_95]]
  (1) INVALID_ARGUMENT:  assertion failed: [loop must iterate at least once to initialize outputs]
         [[{{node Assert/Assert}}]]
         [[while/body/_1/gradient_tape/while/sequential/global_average_pooling2d/ones/_86]]
  (2) INVALID_ARGUMENT:  assertion failed: [loop must iterate at least once to initialize outputs]
         [[{{node Assert/Assert}}]]
0 successful operations.
0 derived errors ignored. [Op:__inference_train_function_5209]

In fact, using with tf.device('/gpu:1'): instead of with strategy leads to the same error. However, I can remove the error by adding

os.environ["CUDA_VISIBLE_DEVICES"]="1"
...
with tf.device('/GPU:1'):

So I'm guessing (at least in my case), that there's something going on with multiple GPUs more generally and MirroredStrategy just isn't catching the errors.

After lots of headache and different attempts I am suspecting that it actually has to do with the cuda architecture in use and the OS version / driver version / linux kernel version.

In my case, nothing helped to get multiple GPUs to work other than updating to Ubuntu 22.04, ensuring that a new linux kernel 5.15 is running and having CUDA >= 11.3 + updating to latest driver.

I guess for new(er) GPUs, as mentioned I was using 3x RTX A6000, thus Ampere generation NVIDIA GPUs, there is something that simply breaks with Ubuntu 20.04 and/or the old linux kernel available in aforementioned Ubuntu. Even an Ubuntu 22.04 in a docker environment was not helping.

So TL;DR:
For me updating to a more modern Ubuntu, updating linux kernel and then using CUDA >= 11.3 and latest nvidia driver worked.

Any other attempt, like different CUDA versions, driver versions, with or without docker, with Anaconda, ... did not work under Ubuntu 20.04 and older linux kernel with new NVIDIA Ampere cards.

@jiayugedede @jurahul @lcorcodilos which OS + linux kernel + CUDA + driver combination where you using? Could you test if updating to new(er) OS + linux kernel resolves the issue for you, too?

This could be a nccl problem.

https://github.com/NVIDIA/nccl-tests

Try

NCCL_DEBUG=INFO ./build/all_reduce_perf -b 8 -e 128M -f 2 -g 3

and you will get the same result

Is there a solution to this? Running into the same issue and all my packages are upgraded as suggested.

Assigning to @rohan100jain to find the right owner for this issue.

I had an issue similar to this, and i fixed it by disabling IOMMU in the bios. This is relevant if you have an AMD CPU and using Data Parallel

I got around the issue for now by modifying the cross_device_ops to tf.distribute.HierarchicalCopyAllReduce() as per stackoverflow . Full code snippet:

strategy = tf.distribute.MirroredStrategy( cross_device_ops=tf.distribute.HierarchicalCopyAllReduce())

I still think this is an issue with the nccl, because the default argument for that parameter is tf.distribute.NcclAllReduce