Error when running akita_train.py #144
Comments
|
Hi, this indicates a mismatch between the sequence lengths, data resolution, and model pooling. If you send the statistics.json file of your dataset and you’re parameters json file of your model, I can help debug. |
|
Thank you for your reply. I can't access the files right now but I think it is the same as what the tutorial generates cause I didn't change anything in it. I can send you mine in a few days if you want. |
|
Hi, I tracked down the bug. Pull the latest from master branch and rerun the notebook from the beginning, including the dataset generation. |
|
Hello, I also encountered this problem, how to debug this problem |
|
Hi, can you share some details about how you're running the script and the error output that you see? I thought I fixed this bug. |
|
Thank U. This is the err_log.
Sincerely, Wen
At 2023-03-24 06:11:27, "David Kelley" ***@***.***> wrote:
Hi, can you share some details about how you're running the script and the error output that you see? I thought I fixed this bug.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***>
python basenji_train.py -k -o ./data/1m/train_out/ ./data/1m/params_tutorial.json ./data/1m/
2023-03-25 12:57:32.355071: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
2023-03-25 12:57:32.571997: I tensorflow/core/util/port.cc:104] oneDNN custom operations are on. You may see slightly different numerical results due to floating-point round-off errors from different computation orders. To turn them off, set the environment variable `TF_ENABLE_ONEDNN_OPTS=0`.
2023-03-25 12:57:32.575328: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-03-25 12:57:32.575342: I tensorflow/compiler/xla/stream_executor/cuda/cudart_stub.cc:29] Ignore above cudart dlerror if you do not have a GPU set up on your machine.
2023-03-25 12:57:33.961203: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer.so.7'; dlerror: libnvinfer.so.7: cannot open shared object file: No such file or directory
2023-03-25 12:57:33.961325: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libnvinfer_plugin.so.7'; dlerror: libnvinfer_plugin.so.7: cannot open shared object file: No such file or directory
2023-03-25 12:57:33.961344: W tensorflow/compiler/tf2tensorrt/utils/py_utils.cc:38] TF-TRT Warning: Cannot dlopen some TensorRT libraries. If you would like to use Nvidia GPU with TensorRT, please make sure the missing libraries mentioned above are installed properly.
2023-03-25 12:57:35.935947: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudart.so.11.0'; dlerror: libcudart.so.11.0: cannot open shared object file: No such file or directory
2023-03-25 12:57:35.936030: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublas.so.11'; dlerror: libcublas.so.11: cannot open shared object file: No such file or directory
2023-03-25 12:57:35.936081: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcublasLt.so.11'; dlerror: libcublasLt.so.11: cannot open shared object file: No such file or directory
2023-03-25 12:57:35.936120: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcufft.so.10'; dlerror: libcufft.so.10: cannot open shared object file: No such file or directory
2023-03-25 12:57:35.991712: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcusparse.so.11'; dlerror: libcusparse.so.11: cannot open shared object file: No such file or directory
2023-03-25 12:57:35.991776: W tensorflow/compiler/xla/stream_executor/platform/default/dso_loader.cc:64] Could not load dynamic library 'libcudnn.so.8'; dlerror: libcudnn.so.8: cannot open shared object file: No such file or directory
2023-03-25 12:57:35.991802: W tensorflow/core/common_runtime/gpu/gpu_device.cc:1934] Cannot dlopen some GPU libraries. Please make sure the missing libraries mentioned above are installed properly if you would like to use GPU. Follow the guide at https://www.tensorflow.org/install/gpu for how to download and setup the required libraries for your platform.
Skipping registering GPU devices...
2023-03-25 12:57:35.992460: I tensorflow/core/platform/cpu_feature_guard.cc:193] This TensorFlow binary is optimized with oneAPI Deep Neural Network Library (oneDNN) to use the following CPU instructions in performance-critical operations: AVX2 AVX512F AVX512_VNNI FMA
To enable them in other operations, rebuild TensorFlow with the appropriate compiler flags.
WARNING:tensorflow:From /data/user/liangbw/anaconda3/envs/basenji/lib/python3.8/site-packages/tensorflow/python/autograph/pyct/static_analysis/liveness.py:83: Analyzer.lamba_check (from tensorflow.python.autograph.pyct.static_analysis.liveness) is deprecated and will be removed after 2023-09-23.
Instructions for updating:
Lambda fuctions will be no more assumed to be used in the statement where they are used, or at least in the same block. tensorflow/tensorflow#56089
Model: "model_1"
…__________________________________________________________________________________________________
Layer (type) Output Shape Param # Connected to
==================================================================================================
sequence (InputLayer) [(None, 1048576, 4) 0 []
]
stochastic_reverse_complement ((None, 1048576, 4) 0 ['sequence[0][0]']
(StochasticReverseComplement) , ())
stochastic_shift (StochasticSh (None, 1048576, 4) 0 ['stochastic_reverse_complement[0
ift) ][0]']
re_lu (ReLU) (None, 1048576, 4) 0 ['stochastic_shift[0][0]']
conv1d (Conv1D) (None, 1048576, 96) 4224 ['re_lu[0][0]']
batch_normalization (BatchNorm (None, 1048576, 96) 384 ['conv1d[0][0]']
alization)
max_pooling1d (MaxPooling1D) (None, 524288, 96) 0 ['batch_normalization[0][0]']
re_lu_1 (ReLU) (None, 524288, 96) 0 ['max_pooling1d[0][0]']
conv1d_1 (Conv1D) (None, 524288, 96) 46080 ['re_lu_1[0][0]']
batch_normalization_1 (BatchNo (None, 524288, 96) 384 ['conv1d_1[0][0]']
rmalization)
max_pooling1d_1 (MaxPooling1D) (None, 262144, 96) 0 ['batch_normalization_1[0][0]']
re_lu_2 (ReLU) (None, 262144, 96) 0 ['max_pooling1d_1[0][0]']
conv1d_2 (Conv1D) (None, 262144, 96) 46080 ['re_lu_2[0][0]']
batch_normalization_2 (BatchNo (None, 262144, 96) 384 ['conv1d_2[0][0]']
rmalization)
max_pooling1d_2 (MaxPooling1D) (None, 131072, 96) 0 ['batch_normalization_2[0][0]']
re_lu_3 (ReLU) (None, 131072, 96) 0 ['max_pooling1d_2[0][0]']
conv1d_3 (Conv1D) (None, 131072, 96) 46080 ['re_lu_3[0][0]']
batch_normalization_3 (BatchNo (None, 131072, 96) 384 ['conv1d_3[0][0]']
rmalization)
max_pooling1d_3 (MaxPooling1D) (None, 65536, 96) 0 ['batch_normalization_3[0][0]']
re_lu_4 (ReLU) (None, 65536, 96) 0 ['max_pooling1d_3[0][0]']
conv1d_4 (Conv1D) (None, 65536, 96) 46080 ['re_lu_4[0][0]']
batch_normalization_4 (BatchNo (None, 65536, 96) 384 ['conv1d_4[0][0]']
rmalization)
max_pooling1d_4 (MaxPooling1D) (None, 32768, 96) 0 ['batch_normalization_4[0][0]']
re_lu_5 (ReLU) (None, 32768, 96) 0 ['max_pooling1d_4[0][0]']
conv1d_5 (Conv1D) (None, 32768, 96) 46080 ['re_lu_5[0][0]']
batch_normalization_5 (BatchNo (None, 32768, 96) 384 ['conv1d_5[0][0]']
rmalization)
max_pooling1d_5 (MaxPooling1D) (None, 16384, 96) 0 ['batch_normalization_5[0][0]']
re_lu_6 (ReLU) (None, 16384, 96) 0 ['max_pooling1d_5[0][0]']
conv1d_6 (Conv1D) (None, 16384, 96) 46080 ['re_lu_6[0][0]']
batch_normalization_6 (BatchNo (None, 16384, 96) 384 ['conv1d_6[0][0]']
rmalization)
max_pooling1d_6 (MaxPooling1D) (None, 8192, 96) 0 ['batch_normalization_6[0][0]']
re_lu_7 (ReLU) (None, 8192, 96) 0 ['max_pooling1d_6[0][0]']
conv1d_7 (Conv1D) (None, 8192, 96) 46080 ['re_lu_7[0][0]']
batch_normalization_7 (BatchNo (None, 8192, 96) 384 ['conv1d_7[0][0]']
rmalization)
max_pooling1d_7 (MaxPooling1D) (None, 4096, 96) 0 ['batch_normalization_7[0][0]']
re_lu_8 (ReLU) (None, 4096, 96) 0 ['max_pooling1d_7[0][0]']
conv1d_8 (Conv1D) (None, 4096, 96) 46080 ['re_lu_8[0][0]']
batch_normalization_8 (BatchNo (None, 4096, 96) 384 ['conv1d_8[0][0]']
rmalization)
max_pooling1d_8 (MaxPooling1D) (None, 2048, 96) 0 ['batch_normalization_8[0][0]']
re_lu_9 (ReLU) (None, 2048, 96) 0 ['max_pooling1d_8[0][0]']
conv1d_9 (Conv1D) (None, 2048, 96) 46080 ['re_lu_9[0][0]']
batch_normalization_9 (BatchNo (None, 2048, 96) 384 ['conv1d_9[0][0]']
rmalization)
max_pooling1d_9 (MaxPooling1D) (None, 1024, 96) 0 ['batch_normalization_9[0][0]']
re_lu_10 (ReLU) (None, 1024, 96) 0 ['max_pooling1d_9[0][0]']
conv1d_10 (Conv1D) (None, 1024, 96) 46080 ['re_lu_10[0][0]']
batch_normalization_10 (BatchN (None, 1024, 96) 384 ['conv1d_10[0][0]']
ormalization)
max_pooling1d_10 (MaxPooling1D (None, 512, 96) 0 ['batch_normalization_10[0][0]']
)
re_lu_11 (ReLU) (None, 512, 96) 0 ['max_pooling1d_10[0][0]']
conv1d_11 (Conv1D) (None, 512, 48) 13824 ['re_lu_11[0][0]']
batch_normalization_11 (BatchN (None, 512, 48) 192 ['conv1d_11[0][0]']
ormalization)
re_lu_12 (ReLU) (None, 512, 48) 0 ['batch_normalization_11[0][0]']
conv1d_12 (Conv1D) (None, 512, 96) 4608 ['re_lu_12[0][0]']
batch_normalization_12 (BatchN (None, 512, 96) 384 ['conv1d_12[0][0]']
ormalization)
dropout (Dropout) (None, 512, 96) 0 ['batch_normalization_12[0][0]']
add (Add) (None, 512, 96) 0 ['max_pooling1d_10[0][0]',
'dropout[0][0]']
re_lu_13 (ReLU) (None, 512, 96) 0 ['add[0][0]']
conv1d_13 (Conv1D) (None, 512, 48) 13824 ['re_lu_13[0][0]']
batch_normalization_13 (BatchN (None, 512, 48) 192 ['conv1d_13[0][0]']
ormalization)
re_lu_14 (ReLU) (None, 512, 48) 0 ['batch_normalization_13[0][0]']
conv1d_14 (Conv1D) (None, 512, 96) 4608 ['re_lu_14[0][0]']
batch_normalization_14 (BatchN (None, 512, 96) 384 ['conv1d_14[0][0]']
ormalization)
dropout_1 (Dropout) (None, 512, 96) 0 ['batch_normalization_14[0][0]']
add_1 (Add) (None, 512, 96) 0 ['add[0][0]',
'dropout_1[0][0]']
re_lu_15 (ReLU) (None, 512, 96) 0 ['add_1[0][0]']
conv1d_15 (Conv1D) (None, 512, 48) 13824 ['re_lu_15[0][0]']
batch_normalization_15 (BatchN (None, 512, 48) 192 ['conv1d_15[0][0]']
ormalization)
re_lu_16 (ReLU) (None, 512, 48) 0 ['batch_normalization_15[0][0]']
conv1d_16 (Conv1D) (None, 512, 96) 4608 ['re_lu_16[0][0]']
batch_normalization_16 (BatchN (None, 512, 96) 384 ['conv1d_16[0][0]']
ormalization)
dropout_2 (Dropout) (None, 512, 96) 0 ['batch_normalization_16[0][0]']
add_2 (Add) (None, 512, 96) 0 ['add_1[0][0]',
'dropout_2[0][0]']
re_lu_17 (ReLU) (None, 512, 96) 0 ['add_2[0][0]']
conv1d_17 (Conv1D) (None, 512, 48) 13824 ['re_lu_17[0][0]']
batch_normalization_17 (BatchN (None, 512, 48) 192 ['conv1d_17[0][0]']
ormalization)
re_lu_18 (ReLU) (None, 512, 48) 0 ['batch_normalization_17[0][0]']
conv1d_18 (Conv1D) (None, 512, 96) 4608 ['re_lu_18[0][0]']
batch_normalization_18 (BatchN (None, 512, 96) 384 ['conv1d_18[0][0]']
ormalization)
dropout_3 (Dropout) (None, 512, 96) 0 ['batch_normalization_18[0][0]']
add_3 (Add) (None, 512, 96) 0 ['add_2[0][0]',
'dropout_3[0][0]']
re_lu_19 (ReLU) (None, 512, 96) 0 ['add_3[0][0]']
conv1d_19 (Conv1D) (None, 512, 48) 13824 ['re_lu_19[0][0]']
batch_normalization_19 (BatchN (None, 512, 48) 192 ['conv1d_19[0][0]']
ormalization)
re_lu_20 (ReLU) (None, 512, 48) 0 ['batch_normalization_19[0][0]']
conv1d_20 (Conv1D) (None, 512, 96) 4608 ['re_lu_20[0][0]']
batch_normalization_20 (BatchN (None, 512, 96) 384 ['conv1d_20[0][0]']
ormalization)
dropout_4 (Dropout) (None, 512, 96) 0 ['batch_normalization_20[0][0]']
add_4 (Add) (None, 512, 96) 0 ['add_3[0][0]',
'dropout_4[0][0]']
re_lu_21 (ReLU) (None, 512, 96) 0 ['add_4[0][0]']
conv1d_21 (Conv1D) (None, 512, 48) 13824 ['re_lu_21[0][0]']
batch_normalization_21 (BatchN (None, 512, 48) 192 ['conv1d_21[0][0]']
ormalization)
re_lu_22 (ReLU) (None, 512, 48) 0 ['batch_normalization_21[0][0]']
conv1d_22 (Conv1D) (None, 512, 96) 4608 ['re_lu_22[0][0]']
batch_normalization_22 (BatchN (None, 512, 96) 384 ['conv1d_22[0][0]']
ormalization)
dropout_5 (Dropout) (None, 512, 96) 0 ['batch_normalization_22[0][0]']
add_5 (Add) (None, 512, 96) 0 ['add_4[0][0]',
'dropout_5[0][0]']
re_lu_23 (ReLU) (None, 512, 96) 0 ['add_5[0][0]']
conv1d_23 (Conv1D) (None, 512, 48) 13824 ['re_lu_23[0][0]']
batch_normalization_23 (BatchN (None, 512, 48) 192 ['conv1d_23[0][0]']
ormalization)
re_lu_24 (ReLU) (None, 512, 48) 0 ['batch_normalization_23[0][0]']
conv1d_24 (Conv1D) (None, 512, 96) 4608 ['re_lu_24[0][0]']
batch_normalization_24 (BatchN (None, 512, 96) 384 ['conv1d_24[0][0]']
ormalization)
dropout_6 (Dropout) (None, 512, 96) 0 ['batch_normalization_24[0][0]']
add_6 (Add) (None, 512, 96) 0 ['add_5[0][0]',
'dropout_6[0][0]']
re_lu_25 (ReLU) (None, 512, 96) 0 ['add_6[0][0]']
conv1d_25 (Conv1D) (None, 512, 48) 13824 ['re_lu_25[0][0]']
batch_normalization_25 (BatchN (None, 512, 48) 192 ['conv1d_25[0][0]']
ormalization)
re_lu_26 (ReLU) (None, 512, 48) 0 ['batch_normalization_25[0][0]']
conv1d_26 (Conv1D) (None, 512, 96) 4608 ['re_lu_26[0][0]']
batch_normalization_26 (BatchN (None, 512, 96) 384 ['conv1d_26[0][0]']
ormalization)
dropout_7 (Dropout) (None, 512, 96) 0 ['batch_normalization_26[0][0]']
add_7 (Add) (None, 512, 96) 0 ['add_6[0][0]',
'dropout_7[0][0]']
re_lu_27 (ReLU) (None, 512, 96) 0 ['add_7[0][0]']
conv1d_27 (Conv1D) (None, 512, 64) 30720 ['re_lu_27[0][0]']
batch_normalization_27 (BatchN (None, 512, 64) 256 ['conv1d_27[0][0]']
ormalization)
re_lu_28 (ReLU) (None, 512, 64) 0 ['batch_normalization_27[0][0]']
one_to_two (OneToTwo) (None, 512, 512, 64 0 ['re_lu_28[0][0]']
)
concat_dist2d (ConcatDist2D) (None, 512, 512, 65 0 ['one_to_two[0][0]']
)
re_lu_29 (ReLU) (None, 512, 512, 65 0 ['concat_dist2d[0][0]']
)
conv2d (Conv2D) (None, 512, 512, 48 28080 ['re_lu_29[0][0]']
)
batch_normalization_28 (BatchN (None, 512, 512, 48 192 ['conv2d[0][0]']
ormalization) )
symmetrize2d (Symmetrize2D) (None, 512, 512, 48 0 ['batch_normalization_28[0][0]']
)
re_lu_30 (ReLU) (None, 512, 512, 48 0 ['symmetrize2d[0][0]']
)
conv2d_1 (Conv2D) (None, 512, 512, 24 10368 ['re_lu_30[0][0]']
)
batch_normalization_29 (BatchN (None, 512, 512, 24 96 ['conv2d_1[0][0]']
ormalization) )
re_lu_31 (ReLU) (None, 512, 512, 24 0 ['batch_normalization_29[0][0]']
)
conv2d_2 (Conv2D) (None, 512, 512, 48 1152 ['re_lu_31[0][0]']
)
batch_normalization_30 (BatchN (None, 512, 512, 48 192 ['conv2d_2[0][0]']
ormalization) )
dropout_8 (Dropout) (None, 512, 512, 48 0 ['batch_normalization_30[0][0]']
)
add_8 (Add) (None, 512, 512, 48 0 ['symmetrize2d[0][0]',
) 'dropout_8[0][0]']
symmetrize2d_1 (Symmetrize2D) (None, 512, 512, 48 0 ['add_8[0][0]']
)
re_lu_32 (ReLU) (None, 512, 512, 48 0 ['symmetrize2d_1[0][0]']
)
conv2d_3 (Conv2D) (None, 512, 512, 24 10368 ['re_lu_32[0][0]']
)
batch_normalization_31 (BatchN (None, 512, 512, 24 96 ['conv2d_3[0][0]']
ormalization) )
re_lu_33 (ReLU) (None, 512, 512, 24 0 ['batch_normalization_31[0][0]']
)
conv2d_4 (Conv2D) (None, 512, 512, 48 1152 ['re_lu_33[0][0]']
)
batch_normalization_32 (BatchN (None, 512, 512, 48 192 ['conv2d_4[0][0]']
ormalization) )
dropout_9 (Dropout) (None, 512, 512, 48 0 ['batch_normalization_32[0][0]']
)
add_9 (Add) (None, 512, 512, 48 0 ['symmetrize2d_1[0][0]',
) 'dropout_9[0][0]']
symmetrize2d_2 (Symmetrize2D) (None, 512, 512, 48 0 ['add_9[0][0]']
)
re_lu_34 (ReLU) (None, 512, 512, 48 0 ['symmetrize2d_2[0][0]']
)
conv2d_5 (Conv2D) (None, 512, 512, 24 10368 ['re_lu_34[0][0]']
)
batch_normalization_33 (BatchN (None, 512, 512, 24 96 ['conv2d_5[0][0]']
ormalization) )
re_lu_35 (ReLU) (None, 512, 512, 24 0 ['batch_normalization_33[0][0]']
)
conv2d_6 (Conv2D) (None, 512, 512, 48 1152 ['re_lu_35[0][0]']
)
batch_normalization_34 (BatchN (None, 512, 512, 48 192 ['conv2d_6[0][0]']
ormalization) )
dropout_10 (Dropout) (None, 512, 512, 48 0 ['batch_normalization_34[0][0]']
)
add_10 (Add) (None, 512, 512, 48 0 ['symmetrize2d_2[0][0]',
) 'dropout_10[0][0]']
symmetrize2d_3 (Symmetrize2D) (None, 512, 512, 48 0 ['add_10[0][0]']
)
re_lu_36 (ReLU) (None, 512, 512, 48 0 ['symmetrize2d_3[0][0]']
)
conv2d_7 (Conv2D) (None, 512, 512, 24 10368 ['re_lu_36[0][0]']
)
batch_normalization_35 (BatchN (None, 512, 512, 24 96 ['conv2d_7[0][0]']
ormalization) )
re_lu_37 (ReLU) (None, 512, 512, 24 0 ['batch_normalization_35[0][0]']
)
conv2d_8 (Conv2D) (None, 512, 512, 48 1152 ['re_lu_37[0][0]']
)
batch_normalization_36 (BatchN (None, 512, 512, 48 192 ['conv2d_8[0][0]']
ormalization) )
dropout_11 (Dropout) (None, 512, 512, 48 0 ['batch_normalization_36[0][0]']
)
add_11 (Add) (None, 512, 512, 48 0 ['symmetrize2d_3[0][0]',
) 'dropout_11[0][0]']
symmetrize2d_4 (Symmetrize2D) (None, 512, 512, 48 0 ['add_11[0][0]']
)
re_lu_38 (ReLU) (None, 512, 512, 48 0 ['symmetrize2d_4[0][0]']
)
conv2d_9 (Conv2D) (None, 512, 512, 24 10368 ['re_lu_38[0][0]']
)
batch_normalization_37 (BatchN (None, 512, 512, 24 96 ['conv2d_9[0][0]']
ormalization) )
re_lu_39 (ReLU) (None, 512, 512, 24 0 ['batch_normalization_37[0][0]']
)
conv2d_10 (Conv2D) (None, 512, 512, 48 1152 ['re_lu_39[0][0]']
)
batch_normalization_38 (BatchN (None, 512, 512, 48 192 ['conv2d_10[0][0]']
ormalization) )
dropout_12 (Dropout) (None, 512, 512, 48 0 ['batch_normalization_38[0][0]']
)
add_12 (Add) (None, 512, 512, 48 0 ['symmetrize2d_4[0][0]',
) 'dropout_12[0][0]']
symmetrize2d_5 (Symmetrize2D) (None, 512, 512, 48 0 ['add_12[0][0]']
)
re_lu_40 (ReLU) (None, 512, 512, 48 0 ['symmetrize2d_5[0][0]']
)
conv2d_11 (Conv2D) (None, 512, 512, 24 10368 ['re_lu_40[0][0]']
)
batch_normalization_39 (BatchN (None, 512, 512, 24 96 ['conv2d_11[0][0]']
ormalization) )
re_lu_41 (ReLU) (None, 512, 512, 24 0 ['batch_normalization_39[0][0]']
)
conv2d_12 (Conv2D) (None, 512, 512, 48 1152 ['re_lu_41[0][0]']
)
batch_normalization_40 (BatchN (None, 512, 512, 48 192 ['conv2d_12[0][0]']
ormalization) )
dropout_13 (Dropout) (None, 512, 512, 48 0 ['batch_normalization_40[0][0]']
)
add_13 (Add) (None, 512, 512, 48 0 ['symmetrize2d_5[0][0]',
) 'dropout_13[0][0]']
symmetrize2d_6 (Symmetrize2D) (None, 512, 512, 48 0 ['add_13[0][0]']
)
cropping2d (Cropping2D) (None, 448, 448, 48 0 ['symmetrize2d_6[0][0]']
)
upper_tri (UpperTri) (None, 99681, 48) 0 ['cropping2d[0][0]']
dense (Dense) (None, 99681, 2) 98 ['upper_tri[0][0]']
switch_reverse_triu (SwitchRev (None, 99681, 2) 0 ['dense[0][0]',
erseTriu) 'stochastic_reverse_complement[0
][1]']
==================================================================================================
Total params: 751,506
Trainable params: 746,002
Non-trainable params: 5,504
__________________________________________________________________________________________________
None
model_strides [2048]
target_lengths [99681]
target_crops [-49585]
<basenji.seqnn.SeqNN object at 0x7f4c3e99bd90>
Epoch 1/10000
Traceback (most recent call last):
File "basenji_train.py", line 183, in <module>
main()
File "basenji_train.py", line 172, in main
seqnn_trainer.fit_keras(seqnn_model)
File "/data/user/liangbw/code_L/basenji_proj/basenji/trainer.py", line 139, in fit_keras
seqnn_model.model.fit(
File "/data/user/liangbw/anaconda3/envs/basenji/lib/python3.8/site-packages/keras/utils/traceback_utils.py", line 70, in error_handler
raise e.with_traceback(filtered_tb) from None
File "/data/user/liangbw/anaconda3/envs/basenji/lib/python3.8/site-packages/tensorflow/python/eager/execute.py", line 52, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
tensorflow.python.framework.errors_impl.InvalidArgumentError: Graph execution error:
Input to reshape is a tensor with 498405 values, but the requested shape has 199362
[[{{node Reshape}}]]
[[IteratorGetNext]] [Op:__inference_train_function_22364]
|
|
I can't reproduce the error. Can you make sure you've pulled the latest code from master and cleared out all of the data so that you're starting from scratch? |
|
Dear Prof. David Kelley,
I am sorry but when I try it again using the new code from the github, I am also get this error.
I'm unfamiliar with tensorflow, but I feel like it should be a network input dimension error, and I wonder if there are updates to the data used, etc. Thanks!
Yours
Wen. L.
At 2023-03-26 07:28:20, "David Kelley" ***@***.***> wrote:
I can't reproduce the error. Can you make sure you've pulled the latest code from master and cleared out all of the data so that you're starting from scratch?
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you commented.Message ID: ***@***.***>
|
|
Hmm that's puzzling. What version of Tensorflow are you using? |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Hi,
I'm trying to train a new akita model. But when I followed the tutorial with exactly the same parameters, I got the following error when running akita_train.py :
I built the environment with conda and prespecified.yml on ubuntu 20.04, cuda 11.4, cudnn 8.4.0
How can I deal with it? Thank you
The text was updated successfully, but these errors were encountered: