[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support synchronous training with parameter servers using Distribution Strategies #27725

Open
yuefengz opened this issue Apr 10, 2019 · 7 comments
Open

Support synchronous training with parameter servers using Distribution Strategies #27725

yuefengz opened this issue Apr 10, 2019 · 7 comments
Assignees
@yuefengz
Labels
comp:dist-strat Distribution Strategy related issues type:feature Feature requests

Comments

@yuefengz
Copy link
Contributor
yuefengz commented Apr 10, 2019 •
edited
Loading

System information

  • TensorFlow version (you are using): TF 2.0
  • Are you willing to contribute it (Yes/No): Yes

Describe the feature and the current behavior/state.
Right now we have MirroredStrategy and MultiWorkerMirroredStrategy for synchronous training where variables and ops are all replicated and all-reduce is used for gradient aggregation. We also have ParameterServerStrategy where gradient updates from workers are purely asynchronous since SyncReplicasOptimizer is buggy and has been deprecated.

We would like to first collect use cases where synchronous training with MirroredStrategy and MultiWorkerMirroredStrategy is not ideal and synchronous training with parameter servers is necessary.

If this feature is necessary and important enough, we will then use this issue to track the progress of the development of this feature.

We have a separate feature request to support large embeddings with MirroredStrategy and MultiWorkerMirroredStrategy: #27726

Will this change the current api? How?
Yes.

Who will benefit with this feature?
Those who use distributed training.

Any Other info.
N/A
@yuefengz yuefengz self-assigned this Apr 10, 2019
@josh11b josh11b added comp:dist-strat Distribution Strategy related issues type:feature Feature requests labels Apr 10, 2019
@shendiaomo
Copy link

What's the progress of this issue?

1 similar comment
@liyi193328
Copy link

What's the progress of this issue?

@neuzxy
Copy link
neuzxy commented Feb 21, 2020

This feature is promising. What's the progress of this issue? @yuefengz

@yuefengz
Copy link
Contributor Author

We will send out a RFC for PSStrategy soon.

@liyi193328
Copy link

There is any plan? Thanks @yuefengz

@sahiltyagi4
Copy link

@yuefengz any developments on this ?

@Mesilenceki
Copy link

hi , Is there some progress with this issue?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@yuefengz yuefengz

Labels
comp:dist-strat Distribution Strategy related issues type:feature Feature requests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@yuefengz @neuzxy @shendiaomo @liyi193328 @sahiltyagi4 @josh11b @Mesilenceki