The problem is more important now with the introduction of new distribution strategies like MultiWorkerMirroredStrategy.

Before with PS strategy we were able to start the tf server on our own:

 server = tf.train.Server(
            tf.train.ClusterSpec(cluster_spec),
            job_name=task_type,
            task_index=task_id,
            config=session_config,
            start=True)

and then just injecting 'google' env in here to prevent the server from starting again:
https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/distribute/distribute_coordinator.py#L432

This reduced the race condition of port reservation and therefore we never really saw the problem.

Now with MultiWorkerMirroredStrategy we can't start the server upfront anymore but need to start the whole strategy which then starts the server which leaves a longer delay between port reservation and the port really taken.
(See https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/distribute/collective_all_reduce_strategy.py#L278)

@superbobry

I am curious why you want to start the servers upfront? Why not allow distribution strategies create std TF servers?

Not being able to reserve a port is really a Yarn issue, not a TF one.

Currently TF server does not support clean shutdown, and as a workaround I'd suggest you to use a dummy server to probe available ports instead of creating and destroying TF servers repeatedly.

We currently use the solution described by @byronyi. But we encounter some race conditions with it. Because of the option SO_REUSE_PORT is not set, we have to shutdown the dummy server before to start the TF sever. As yarn cannot reserve ports, some other rogue processes can take the port before the tensorflow server. For this reason, we would like to activate this option to be sure the port will be reserved to tensorflow.

@jdlesage I see what you mean by the race condition. Unfortunately, using SO_REUSEPORT does not solve this issue. Suppose two independent TF job bind to the same port with SO_REUSEPORT on a same machine. They thought they could reserve a port, but since both of them are using SO_REUSEPORT, neither of them will think they fail.

You have to start the servers upfront and propagate the cluster spec after the server initialization.

I was not aware that it is possible to propagate the cluster spec after the servers initialization. Definitively, that's the best solution. Do you have some pointers that describe how to propagate the cluster spec ?

Take a look at this: #11081

Not sure how this will work with dist-strat though; I will leave the question to @yuefengz.

Passing a ClusterResolver instead of TF_CONFIG could be helpful as well.

Thanks for the replies.

@byronyi
I had a look at using ClusterResolver instead of TF_CONFIG but it seems that this doesn't solve this problem. From what I understood all it does is giving the spec in some form to dist strategies which then start the server again. So I suppose the race condition would be the same.

The code you linked to is nice and we actually use this for low level tensorflow in tf 1.15 but it doesn't seem to work anymore with tf 2 (at least all methods display as deprecated)

I see what you mean by the race condition. Unfortunately, using SO_REUSEPORT does not solve this issue. Suppose two independent TF job bind to the same port with SO_REUSEPORT on a same machine. They thought they could reserve a port, but since both of them are using SO_REUSEPORT, neither of them will think they fail.

I don't necessarily agree on this. We can have two tf jobs starting at the same time, yes, but the initial port assignment is random, so the probability that this collides is really small. Also in this case we could even code a port assignment service on our own and only reserve free ports.
With the current tf behavior we are blocked because we need to free the ports all the time (before starting the tf server) and then we don't even know when tf assigns them again.
With PS strategy this timeframe is really small, because we can start the server upfront, so we have this race condition but it is not a real issue.
With CollectiveAllReduce strategy (aka MultiWorkerMirroredStrategy) it is an issue because it seems not possible to start the server upfront anymore (@yuefengz can you confirm this statement is true ?)

So, imo, we would need two fixes:

• re-activate SO_REUSEPORT
• permit in every case to start the tf server upfront.

This is not supported for MultiWorkerMirroredStrategy to skip starting std servers. The server is started by the context object. @haoyuz probably can share a workaround if any.