[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error in running DNNF-GHPR benchmark #14

Open
shenw000 opened this issue Oct 24, 2022 · 1 comment
Open

Error in running DNNF-GHPR benchmark #14

shenw000 opened this issue Oct 24, 2022 · 1 comment

Comments

@shenw000
Copy link

When I run DNNF-GHPR benchmark, I receive the following error (the complete error is list is in the attachment:

dnnf properties/dronet_property_0.py --network N onnx/dronet.onnx --backend cleverhans.projected_gradient_descent
Falsifying: Forall(x, (((0 <= x) & (x <= 1) & (N(slice(2, -3, None), 1) <= -2.1972245773362196)) ==> ((-0.08726646259971647 <= N(slice(2, -1, None), 0)) & (N(slice(2, -1, None), 0) <= 0.08726646259971647))))

Network N:
Input_0 : Input([ -1 200 200 1], dtype=float32)
Transpose_0 : Transpose(Input_0, permutation=[0 3 1 2])
Conv_0 : Conv(Transpose_0, out_channels=32, kernel_shape=[5, 5], strides=[2, 2], pads=[1, 1, 2, 2], group=1, dilations=[1, 1])
MaxPool_0 : MaxPool(Conv_0)
Identity_0 : Identity(MaxPool_0)
Conv_1 : Conv(Identity_0, out_channels=32, kernel_shape=[1, 1], strides=[2, 2], pads=[0, 0, 0, 0], group=1, dilations=[1, 1])
BatchNormalization_0 : BatchNormalization(Identity_0)
Relu_0 : Relu(BatchNormalization_0)
Conv_2 : Conv(Relu_0, out_channels=32, kernel_shape=[3, 3], strides=[2, 2], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
BatchNormalization_1 : BatchNormalization(Conv_2)
Relu_1 : Relu(BatchNormalization_1)
Conv_3 : Conv(Relu_1, out_channels=32, kernel_shape=[3, 3], strides=[1, 1], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
Add_0 : Add(Conv_1, Conv_3)
Conv_4 : Conv(Add_0, out_channels=64, kernel_shape=[1, 1], strides=[2, 2], pads=[0, 0, 0, 0], group=1, dilations=[1, 1])
BatchNormalization_2 : BatchNormalization(Add_0)
Relu_2 : Relu(BatchNormalization_2)
Conv_5 : Conv(Relu_2, out_channels=64, kernel_shape=[3, 3], strides=[2, 2], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
BatchNormalization_3 : BatchNormalization(Conv_5)
Relu_3 : Relu(BatchNormalization_3)
Conv_6 : Conv(Relu_3, out_channels=64, kernel_shape=[3, 3], strides=[1, 1], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
Add_1 : Add(Conv_4, Conv_6)
Conv_7 : Conv(Add_1, out_channels=128, kernel_shape=[1, 1], strides=[2, 2], pads=[0, 0, 0, 0], group=1, dilations=[1, 1])
BatchNormalization_4 : BatchNormalization(Add_1)
Relu_4 : Relu(BatchNormalization_4)
Conv_8 : Conv(Relu_4, out_channels=128, kernel_shape=[3, 3], strides=[2, 2], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
BatchNormalization_5 : BatchNormalization(Conv_8)
Relu_5 : Relu(BatchNormalization_5)
Conv_9 : Conv(Relu_5, out_channels=128, kernel_shape=[3, 3], strides=[1, 1], pads=[1, 1, 1, 1], group=1, dilations=[1, 1])
Add_2 : Add(Conv_7, Conv_9)
Transpose_1 : Transpose(Add_2, permutation=[0 2 3 1])
Flatten_0 : Flatten(Transpose_1, axis=1)
Relu_6 : Relu(Flatten_0)
Identity_1 : Identity(Relu_6)
MatMul_0 : MatMul(Identity_1, ndarray(shape=(6272, 1)))
Add_3 : Add(MatMul_0, [0.01033337])
Identity_2 : Identity(Add_3)
MatMul_1 : MatMul(Identity_1, ndarray(shape=(6272, 1)))
Add_4 : Add(MatMul_1, [1.1624445])
Sigmoid_0 : Sigmoid(Add_4)
Identity_3 : Identity(Sigmoid_0)
Concat_0 : Concat(['Identity_2', 'Identity_3'])

2022-10-24 11:48:55.000739: I tensorflow/core/platform/cpu_feature_guard.cc:151] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2022-10-24 11:48:57.642840: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:0 with 5222 MB memory: -> device: 0, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:3b:00.0, compute capability: 6.1
2022-10-24 11:48:57.644271: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:1 with 7072 MB memory: -> device: 1, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:5e:00.0, compute capability: 6.1
2022-10-24 11:48:57.646563: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:2 with 7096 MB memory: -> device: 2, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:86:00.0, compute capability: 6.1
2022-10-24 11:48:57.649086: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1525] Created device /job:localhost/replica:0/task:0/device:GPU:3 with 7045 MB memory: -> device: 3, name: NVIDIA GeForce GTX 1080 Ti, pci bus id: 0000:af:00.0, compute capability: 6.1
2022-10-24 11:48:58.140240: I tensorflow/stream_executor/cuda/cuda_dnn.cc:366] Loaded cuDNN version 8201
2022-10-24 11:49:00.473526: E tensorflow/stream_executor/cuda/cuda_blas.cc:197] failed to set new cublas math mode: CUBLAS_STATUS_INVALID_VALUE
2022-10-24 11:49:00.473600: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:442 : INTERNAL: Failed initializing math mode
Traceback (most recent call last):
File "/media/sdc1/shenw/sandbox/nn_assurance/dnnf/.venv/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 58, in wrapped_func
self._cache[func] = func(*args, **kwargs)
File "/media/sdc1/shenw/sandbox/nn_assurance/dnnf/.venv/lib/python3.8/site-packages/dnnv/nn/converters/tensorflow.py", line 588, in matmul_func
result = tf.matmul(a, b)

It looks like there is error in
failed to set new cublas math mode: CUBLAS_STATUS_INVALID_VALUE

So I searched the Internet for the solution to this error. And found the following site
tensorflow/tensorflow#57359

A solution is suggested that adding the following line to python code would fix the problem.
tf.config.experimental.enable_tensor_float_32_execution(False)

However I could not locate the tensorflow code in your dnnf codebase, and hence can't add the above line to your code.
Cold you point me to where I could add the line of
tf.config.experimental.enable_tensor_float_32_execution(False)
to fix the error?

Your help will be greatly appreciated. Thanks!

dnnf_ghpr_benchmark_error.txt

@dlshriver
Copy link
Owner

Sorry for the delay, I'm currently busy finishing up my thesis and preparing to defend and haven't been able to do much on this.

TensorFlow is a dependency of DNNV, which DNNF depends on. It is used in several places throughout that project, but primarily in dnnv/nn/converters/tensorflow.py. You should be able to import tensorflow anywhere in DNNF though. Another possible solution could be to try an older version of tensorflow, maybe version 2.2 or 2.3 (e.g., pip install dnnf "tensorflow==2.3").

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants