@david-bernstein Could you please detail your experiment environment such as tf version etc:)

Sure, the tf version varies, but it is usually 1.10.0 or 1.11.0. The code is running on Ubuntu 16.04 and I've run it on machines with a 1070 and 1080Ti GPUs.

This issue with the L2 norm gradient being unstable is known in tensorflow, see tensorflow/tensorflow#12071

I tried to implement the fix described in the post above but am not a TF expert and haven't gotten it to work yet.

@david-bernstein Thanks for your kindly share with us. I will test it later:)

@david-bernstein Thanks for your kindly share with us. I will test it later:)

@MaybeShewill-CV Have you tried this solution? How is the results? Thanks very much!

I also met this problem and I've changed all normalization to L1 norm in lanenet_discriminative_loss.py.
My train accuracy has reached to 97% but my validation accuracy is only 50% after 200k steps. Actually my validation accuracy has been 50% since 100k steps. My global_config.py is unchanged. I wonder if you can give me some suggestions for this situation. Thank you!

@yxxxxxxxx You may first use the model weights to do the testing work on your test dataset to see if the model performs as well as you wish:)

@david-bernstein @yxxxxxxxx I've also met this problem, and change
tf.norm(mu_diff_bool, axis=1) to tf.norm(mu_diff_bool,ord=1, axis=1), but still got nan, how did you do?
thanks!

@yxxxxxxxx There are three calls to tf.norm in lanenet_discriminative_loss.py. I changed all of those to L1.

@david-bernstein
yes, I've changed all of those, as follow:

# 计算公式的loss(var)
distance = tf.norm(tf.subtract(mu_expand, reshaped_pred), ord=1, axis=1)
.........

mu_norm = tf.norm(mu_diff_bool,ord=1, axis=1)
.........

# 
l_reg = tf.reduce_mean(tf.norm(mu, ord=1,axis=1))

whether the operations is right?

@JunJieAI Yeah, I change all L2 normalization to L1. It indeed worked.

I use tensorflow_gpu1.10.0。And I change all L2 normalization to L1,but i got nan loss.I don't know why.

tf.norm(tf.subtract(mu_expand, reshaped_pred), ord=1, axis=1)

@JunJieAI Yeah, I change all L2 normalization to L1. It indeed worked.

Could you please explain me, what changes did you made?