You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
2022-10-24 11:48:55.000739: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-10-24 11:48:57.642840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5222 MB memory: -> device: 0, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:3b:00.0, compute capability: 6.1
2022-10-24 11:48:57.644271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 7072 MB memory: -> device: 1, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:5e:00.0, compute capability: 6.1
2022-10-24 11:48:57.646563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 7096 MB memory: -> device: 2, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:86:00.0, compute capability: 6.1
2022-10-24 11:48:57.649086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 7045 MB memory: -> device: 3, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:af:00.0, compute capability: 6.1
2022-10-24 11:48:58.140240: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
2022-10-24 11:49:00.473526: E tensorflow/stream_executor/cuda/cuda_blas.cc:197] failed to set new cublas math mode: CUBLAS_STATUS_INVALID_VALUE
2022-10-24 11:49:00.473600: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Failed initializing math mode
Traceback (most recent call last):
File "/media/sdc1/shenw/sandbox/nn_assurance/dnnf/.venv/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 58, in wrapped_func
self._cache[func] = func(*args, **kwargs)
File "/media/sdc1/shenw/sandbox/nn_assurance/dnnf/.venv/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 588, in matmul_func
result = tf.matmul(a, b)
It looks like there is error in failed to set new cublas math mode: CUBLAS_STATUS_INVALID_VALUE
So I searched the Internet for the solution to this error. And found the following site tensorflow/tensorflow#57359
A solution is suggested that adding the following line to python code would fix the problem.
tf.config.experimental.enable_tensor_float_32_execution(False)
However I could not locate the tensorflow code in your dnnf codebase, and hence can't add the above line to your code.
Cold you point me to where I could add the line of
tf.config.experimental.enable_tensor_float_32_execution(False)
to fix the error?
Sorry for the delay, I'm currently busy finishing up my thesis and preparing to defend and haven't been able to do much on this.
TensorFlow is a dependency of DNNV, which DNNF depends on. It is used in several places throughout that project, but primarily in dnnv/nn/converters/tensorflow.py. You should be able to import tensorflow anywhere in DNNF though. Another possible solution could be to try an older version of tensorflow, maybe version 2.2 or 2.3 (e.g., pip install dnnf "tensorflow==2.3").
When I run DNNF-GHPR benchmark, I receive the following error (the complete error is list is in the attachment:
dnnf properties/dronet_property_0.py --network N onnx/dronet.onnx --backend cleverhans.projected_gradient_descent
Falsifying: Forall(x, (((0 <= x) & (x <= 1) & (N(slice(2, -3, None), 1) <= -2.1972245773362196)) ==> ((-0.08726646259971647 <= N(slice(2, -1, None), 0)) & (N(slice(2, -1, None), 0) <= 0.08726646259971647))))
Network N:
Input_0 : Input([ -1 200 200 1], dtype=float32)
Transpose_0 : Transpose(Input_0, permutation=[0 3 1 2])
Conv_0 : Conv(Transpose_0, out_channels=32, kernel_shape=[5, 5], strides=[2, 2], pads=[1, 1, 2, 2], group=1, dilations=[1, 1])
MaxPool_0 : MaxPool(Conv_0)
Identity_0 : Identity(MaxPool_0)
Conv_1 : Conv(Identity_0, out_channels=32, kernel_shape=[1, 1], strides=[2, 2], pads=[0, 0, 0, 0], group=1, dilations=[1, 1])
BatchNormalization_0 : BatchNormalization(Identity_0)
Relu_0 : Relu(BatchNormalization_0)
Conv_2 : Conv(Relu_0, out_channels=32, kernel_shape=[3, 3], strides=[2, 2], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
BatchNormalization_1 : BatchNormalization(Conv_2)
Relu_1 : Relu(BatchNormalization_1)
Conv_3 : Conv(Relu_1, out_channels=32, kernel_shape=[3, 3], strides=[1, 1], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
Add_0 : Add(Conv_1, Conv_3)
Conv_4 : Conv(Add_0, out_channels=64, kernel_shape=[1, 1], strides=[2, 2], pads=[0, 0, 0, 0], group=1, dilations=[1, 1])
BatchNormalization_2 : BatchNormalization(Add_0)
Relu_2 : Relu(BatchNormalization_2)
Conv_5 : Conv(Relu_2, out_channels=64, kernel_shape=[3, 3], strides=[2, 2], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
BatchNormalization_3 : BatchNormalization(Conv_5)
Relu_3 : Relu(BatchNormalization_3)
Conv_6 : Conv(Relu_3, out_channels=64, kernel_shape=[3, 3], strides=[1, 1], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
Add_1 : Add(Conv_4, Conv_6)
Conv_7 : Conv(Add_1, out_channels=128, kernel_shape=[1, 1], strides=[2, 2], pads=[0, 0, 0, 0], group=1, dilations=[1, 1])
BatchNormalization_4 : BatchNormalization(Add_1)
Relu_4 : Relu(BatchNormalization_4)
Conv_8 : Conv(Relu_4, out_channels=128, kernel_shape=[3, 3], strides=[2, 2], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
BatchNormalization_5 : BatchNormalization(Conv_8)
Relu_5 : Relu(BatchNormalization_5)
Conv_9 : Conv(Relu_5, out_channels=128, kernel_shape=[3, 3], strides=[1, 1], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
Add_2 : Add(Conv_7, Conv_9)
Transpose_1 : Transpose(Add_2, permutation=[0 2 3 1])
Flatten_0 : Flatten(Transpose_1, axis=1)
Relu_6 : Relu(Flatten_0)
Identity_1 : Identity(Relu_6)
MatMul_0 : MatMul(Identity_1, ndarray(shape=(6272, 1)))
Add_3 : Add(MatMul_0, [0.01033337])
Identity_2 : Identity(Add_3)
MatMul_1 : MatMul(Identity_1, ndarray(shape=(6272, 1)))
Add_4 : Add(MatMul_1, [1.1624445])
Sigmoid_0 : Sigmoid(Add_4)
Identity_3 : Identity(Sigmoid_0)
Concat_0 : Concat(['Identity_2', 'Identity_3'])
2022-10-24 11:48:55.000739: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-10-24 11:48:57.642840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5222 MB memory: -> device: 0, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:3b:00.0, compute capability: 6.1
2022-10-24 11:48:57.644271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 7072 MB memory: -> device: 1, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:5e:00.0, compute capability: 6.1
2022-10-24 11:48:57.646563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 7096 MB memory: -> device: 2, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:86:00.0, compute capability: 6.1
2022-10-24 11:48:57.649086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 7045 MB memory: -> device: 3, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:af:00.0, compute capability: 6.1
2022-10-24 11:48:58.140240: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
2022-10-24 11:49:00.473526: E tensorflow/stream_executor/cuda/cuda_blas.cc:197] failed to set new cublas math mode: CUBLAS_STATUS_INVALID_VALUE
2022-10-24 11:49:00.473600: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Failed initializing math mode
Traceback (most recent call last):
File "/media/sdc1/shenw/sandbox/nn_assurance/dnnf/.venv/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 58, in wrapped_func
self._cache[func] = func(*args, **kwargs)
File "/media/sdc1/shenw/sandbox/nn_assurance/dnnf/.venv/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 588, in matmul_func
result = tf.matmul(a, b)
It looks like there is error in
failed to set new cublas math mode: CUBLAS_STATUS_INVALID_VALUE
So I searched the Internet for the solution to this error. And found the following site
tensorflow/tensorflow#57359
A solution is suggested that adding the following line to python code would fix the problem.
tf.config.experimental.enable_tensor_float_32_execution(False)
However I could not locate the tensorflow code in your dnnf codebase, and hence can't add the above line to your code.
Cold you point me to where I could add the line of
tf.config.experimental.enable_tensor_float_32_execution(False)
to fix the error?
Your help will be greatly appreciated. Thanks!
dnnf_ghpr_benchmark_error.txt
The text was updated successfully, but these errors were encountered: