Numerical precision issue of operators selu, leakyRelu, softplus and their corresponding backward operators on Bfloat16 vs float32 #67440
Labels
comp:ops
OPs related issues
stat:awaiting tensorflower
Status - Awaiting response from tensorflower
TF 2.16
type:bug
Bug
Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
binary
TensorFlow version
tf 2.16.1
Custom code
Yes
OS platform and distribution
No response
Mobile device
No response
Python version
3.10.9
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
I'd like to bring to attention an issue concerning the numerical precision of several operators (selu, leaky, relu) when operating on Bfloat16 versus float32 data types. I conducted comparisons using 20,000 random tensors for these operators, assessing the outputs in both Bfloat16 and float32 and computing the discrepancies. My observations indicate that differences generated by TensorFlow are generally more pronounced compared to PyTorch. Particularly noteworthy is the significant error produced by the SeluGrad operator. The results are summarized in the table below:
In a standalone code to reproduce the issue, I provide illustrative instances for seluGrad operators, where the output discrepancy between Bfloat16 and float32 can be as high as 10.4.
Standalone code to reproduce the issue
Relevant log output
The text was updated successfully, but these errors were encountered: