[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Work-around for preprocessing failing with: INTERNAL: Failed initializing math mode #35

Open
bluenote10 opened this issue Sep 4, 2022 · 0 comments
Open

Work-around for preprocessing failing with: INTERNAL: Failed initializing math mode #35

bluenote10 opened this issue Sep 4, 2022 · 0 comments

Comments

@bluenote10
Copy link

I was trying to run the preprocessing, but it failed with a Tensorflow error at the step when its trying to apply crepe. The error is somewhat cryptic (INTERNAL: Failed initializing math mode) with a similar traceback:

Full traceback 
2022-09-04 14:34:30.720585: E tensorflow/stream_executor/cuda/cuda_blas.cc:197] failed to set new cublas math mode: CUBLAS_STATUS_INVALID_VALUE
2022-09-04 14:34:30.720614: W tensorflow/core/framework/op_kernel.cc:1745] OP_REQUIRES failed at matmul_op_impl.h:438 : INTERNAL: Failed initializing math mode
/home/fabian/Dropbox/Temp/ReferenceAudio/Landola/Snapshot1/recording_normalized.mp3:   0%|                                                                                                                                                                                                       | 0/1 [00:34<?, ?it/s]
Traceback (most recent call last):
  File "preprocess.py", line 91, in <module>
    main()
  File "preprocess.py", line 73, in main
    x, p, l = preprocess(f, **config["preprocess"])
  File "preprocess.py", line 27, in preprocess
    pitch = extract_pitch(x, sampling_rate, block_size)
  File "/home/fabian/git/_ext/ddsp_pytorch/ddsp/core.py", line 101, in extract_pitch
    f0 = crepe.predict(
  File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/crepe/core.py", line 255, in predict
    activation = get_activation(audio, sr, model_capacity=model_capacity,
  File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/crepe/core.py", line 212, in get_activation
    return model.predict(frames, verbose=verbose)
  File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 67, in error_handler
    raise e.with_traceback(filtered_tb) from None
  File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 54, in quick_execute
    tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InternalError: Graph execution error:

Detected at node 'model/classifier/MatMul' defined at (most recent call last):
    File "preprocess.py", line 91, in <module>
      main()
    File "preprocess.py", line 73, in main
      x, p, l = preprocess(f, **config["preprocess"])
    File "preprocess.py", line 27, in preprocess
      pitch = extract_pitch(x, sampling_rate, block_size)
    File "/home/fabian/git/_ext/ddsp_pytorch/ddsp/core.py", line 101, in extract_pitch
      f0 = crepe.predict(
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/crepe/core.py", line 255, in predict
      activation = get_activation(audio, sr, model_capacity=model_capacity,
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/crepe/core.py", line 212, in get_activation
      return model.predict(frames, verbose=verbose)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/engine/training.py", line 2033, in predict
      tmp_batch_outputs = self.predict_function(iterator)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/engine/training.py", line 1845, in predict_function
      return step_function(self, iterator)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/engine/training.py", line 1834, in step_function
      outputs = model.distribute_strategy.run(run_step, args=(data,))
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/engine/training.py", line 1823, in run_step
      outputs = model.predict_step(data)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/engine/training.py", line 1791, in predict_step
      return self(x, training=False)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/engine/training.py", line 490, in __call__
      return super().__call__(*args, **kwargs)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/engine/functional.py", line 458, in call
      return self._run_internal_graph(
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/engine/functional.py", line 596, in _run_internal_graph
      outputs = node.layer(*args, **kwargs)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 64, in error_handler
      return fn(*args, **kwargs)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/engine/base_layer.py", line 1014, in __call__
      outputs = call_fn(inputs, *args, **kwargs)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 92, in error_handler
      return fn(*args, **kwargs)
    File "/home/fabian/.virtualenvs/ddsp_pytorch/lib/python3.8/site-packages/keras/layers/core/dense.py", line 221, in call
      outputs = tf.matmul(a=inputs, b=self.kernel)
Node: 'model/classifier/MatMul'
Failed initializing math mode
	 [[{{node model/classifier/MatMul}}]] [Op:__inference_predict_function_723]

I researched this a bit, and it looks like this relates to the following upstream issue for Tensorflow: tensorflow/tensorflow#57359

I'm mainly opening this issue to inform others of a simple work-around in case they run into this as well. It is possible to simply place a dummy import tensorflow at the beginning of preprocessing.py to avoid the issue. Apparently the issue has something to do with importing pytorch first, followed by tensorflow, which messes up Tensorflow's GPU initialization, and running the Tensorflow import first seems to avoid that problem.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@bluenote10