-
Notifications
You must be signed in to change notification settings - Fork 74k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Feature request: Allow the build to use the system-installed protobuf lib #16104
Comments
Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks. |
sigh for this kind of issue, it really doesn't matter but for the sake of automation, here I go:
|
I think there was a similar question but can't seem to find it. @gunan what's our position here? |
Eventually, we would like to enable this. |
@danbarnes333 yes that's right on. Maybe a stepping stone would be to allow people to link against the system protobuf during the We should also consider, at some point, distributing pip packages that are dependent on a libprotobuf package. |
This would be useful to us as well. We are currently stuck in a situation where the protobuf symbols in libtensorflow_framework.so are being clobbered by symbols in a custom libprotobuf from a different API revision that is being loaded to support the operations of our custom backend to TensorFlow. In other words, we have this library dependency tree right now:
|
We found a workaround by forcing bazel to use a protobuf version that matched the one used elsewhere on the system. Here is a patch that will force usage of the 3.4.0 library, but one can also point to any other archive tag on github. |
@RichDubielzig that is what we have been doing, as well as hiding more symbols in the tensorflow shared library, and has been working ok. However @drpngx mentioned on my original issue that this isnt completely safe unless I misunderstood (about halfway down #8403) but perhaps things have changed since then. Either way simply building against the system protobuf would be great. |
@RichDubielzig Do you mind to tell more detail about how to use the file you gave? Like where should I put it and how can I make it work? Thanks alot. |
If this ever gets picked up you'll probably need protobuf to enable dynamic linking as well - protocolbuffers/protobuf#8291 |
@gnossen gave a great overview in grpc/grpc#24992 of the overall problem. If a python process using both protobuf _and_ another native library linking in libprotobuf frequently can cause crashes. This seems to frequently affect tensorflow as well: tensorflow/tensorflow#8394, tensorflow/tensorflow#9525 (comment) tensorflow/tensorflow#24976, tensorflow/tensorflow#35573, https://github.com/tensorflow/tensorflow/blob/v2.0.0/tensorflow/contrib/makefile/rename_protobuf.sh, tensorflow/tensorflow#16104 Testing locally this fixes both crashes when linking in multiple versions of protobuf and fixes `DescriptorPool` clashes as well (e.g. Python and Native code import different versions of the same message).
@gnossen gave a great overview in grpc/grpc#24992 of the overall problem. If a python process using both protobuf _and_ another native library linking in libprotobuf frequently can cause crashes. This seems to frequently affect tensorflow as well: tensorflow/tensorflow#8394, tensorflow/tensorflow#9525 (comment) tensorflow/tensorflow#24976, tensorflow/tensorflow#35573, https://github.com/tensorflow/tensorflow/blob/v2.0.0/tensorflow/contrib/makefile/rename_protobuf.sh, tensorflow/tensorflow#16104 Testing locally this fixes both crashes when linking in multiple versions of protobuf and fixes `DescriptorPool` clashes as well (e.g. Python and Native code import different versions of the same message).
@gnossen gave a great overview in grpc/grpc#24992 of the overall problem. If a python process using both protobuf _and_ another native library linking in libprotobuf frequently can cause crashes. This seems to frequently affect tensorflow as well: tensorflow/tensorflow#8394, tensorflow/tensorflow#9525 (comment) tensorflow/tensorflow#24976, tensorflow/tensorflow#35573, https://github.com/tensorflow/tensorflow/blob/v2.0.0/tensorflow/contrib/makefile/rename_protobuf.sh, tensorflow/tensorflow#16104 Testing locally this fixes both crashes when linking in multiple versions of protobuf and fixes `DescriptorPool` clashes as well (e.g. Python and Native code import different versions of the same message).
@gnossen gave a great overview in grpc/grpc#24992 of the overall problem. If a python process using both protobuf _and_ another native library linking in libprotobuf frequently can cause crashes. This seems to frequently affect tensorflow as well: tensorflow/tensorflow#8394, tensorflow/tensorflow#9525 (comment) tensorflow/tensorflow#24976, tensorflow/tensorflow#35573, https://github.com/tensorflow/tensorflow/blob/v2.0.0/tensorflow/contrib/makefile/rename_protobuf.sh, tensorflow/tensorflow#16104 Testing locally this fixes both crashes when linking in multiple versions of protobuf and fixes `DescriptorPool` clashes as well (e.g. Python and Native code import different versions of the same message). Co-authored-by: Roy Williams <roy.williams.iii@gmail.com>
This issue is stale because it has been open for 180 days with no activity. It will be closed if no further activity occurs. Thank you. |
This is definitely still valid. |
This issue is stale because it has been open for 180 days with no activity. It will be closed if no further activity occurs. Thank you. |
It's still relevant. |
This issue is stale because it has been open for 180 days with no activity. It will be closed if no further activity occurs. Thank you. |
It's still a problem. |
Describe the problem
Currently, it's impossible to use the system-installed protobuf library because the tensorflow build always uses the
protobuf_archive
version. There should be an option to use the one installed in the system.Background: I package tensorflow for Arch Linux and we run into symbol conflicts if a user wants to use protobuf and tensorflow together in a binary because tensorflow's protobuf symbols conflict with the one installed in the system already.
Original Arch bug report: https://bugs.archlinux.org/task/56943
The text was updated successfully, but these errors were encountered: