[go: nahoru, domu]

Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Invoke get_shape() on sparse_tensor leads to feeding error #13639

Closed
liqimai opened this issue Oct 11, 2017 · 15 comments · Fixed by #69579
Closed

Invoke get_shape() on sparse_tensor leads to feeding error #13639

liqimai opened this issue Oct 11, 2017 · 15 comments · Fixed by #69579
Assignees

Comments

@liqimai
Copy link
liqimai commented Oct 11, 2017

If I invoke get_shape method on sparse_tensor, the shape tensor will be added into the _unfeedable_tensors set of the current graph. Then when I feed the sparse tensor, an error occurs.

The codes below show this error

import tensorflow as tf
import numpy as np

shape = np.array([7, 9, 2], dtype=np.int64)
indices = np.array([[3, 2, 0], [4, 5, 1]], dtype=np.int64)
values = np.array([1.0, 2.0], dtype=np.float32)
x = tf.sparse_placeholder(tf.float32, shape=shape)

with tf.Session() as sess:
    x.get_shape() # <-- Troublemaker
    # This line leads to the exception:
    # 	   Tensor Tensor("Const:0", shape=(3,), dtype=int64) may not be fed.
    # The side effection of this line is that 
    # it adds the 'shape' tensor into Graph._unfeedable_tensors, 

    print(sess.run(x, feed_dict={
    	x: tf.SparseTensorValue(indices, values, shape)}))

The stacktrace

ValueError                                Traceback (most recent call last)
<ipython-input-1-baac6f49a954> in <module>()
     10     x.get_shape()
     11     print(sess.run(x, feed_dict={
---> 12     	x: tf.SparseTensorValue(indices, values, shape)}))

/Users/liqimai/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py in run(self, fetches, feed_dict, options, run_metadata)
    765     try:
    766       result = self._run(None, fetches, feed_dict, options_ptr,
--> 767                          run_metadata_ptr)
    768       if run_metadata:
    769         proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)

/Users/liqimai/anaconda3/lib/python3.5/site-packages/tensorflow/python/client/session.py in _run(self, handle, fetches, feed_dict, options, run_metadata)
    944                 % (np_val.shape, subfeed_t.name, str(subfeed_t.get_shape())))
    945           if not self.graph.is_feedable(subfeed_t):
--> 946             raise ValueError('Tensor %s may not be fed.' % subfeed_t)
    947           subfeed_name = compat.as_bytes(subfeed_t.name)
    948           feed_dict_string[subfeed_name] = np_val

ValueError: Tensor Tensor("Const:0", shape=(3,), dtype=int64) may not be fed.

System information

I do not think this bug is related to my environment.
== cat /etc/issue ===============================================
Darwin liqimaideMacBook-Pro.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64
Mac OS X 10.12.6

== are we in docker =============================================
No

== compiler =====================================================
Apple LLVM version 9.0.0 (clang-900.0.37)
Target: x86_64-apple-darwin16.7.0
Thread model: posix
InstalledDir: /Applications/Xcode.app/Contents/Developer/Toolchains/XcodeDefault.xctoolchain/usr/bin

== uname -a =====================================================
Darwin MacBook-Pro.local 16.7.0 Darwin Kernel Version 16.7.0: Thu Jun 15 17:36:27 PDT 2017; root:xnu-3789.70.16~2/RELEASE_X86_64 x86_64

== check pips ===================================================
numpy (1.11.3)
protobuf (3.2.0)
tensorflow (1.0.0)

== check for virtualenv =========================================
False

== tensorflow import ============================================
tf.VERSION = 1.0.0
tf.GIT_VERSION = v1.0.0-rc2-15-g47bba63-dirty
tf.COMPILER_VERSION = v1.0.0-rc2-15-g47bba63-dirty
Sanity check: array([1], dtype=int32)

== env ==========================================================
LD_LIBRARY_PATH is unset
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
tf_env_collect.sh: line 105: nvidia-smi: command not found

== cuda libs ===================================================

@reedwm
Copy link
Member
reedwm commented Oct 11, 2017

The issue is that SparseTensor.get_shape calls tensor_util.constant_value_as_shape, which calls tensor_util.constant_value, which calls Graph.prevent_feeding.

@ebrevdo, what do you think the best way of solving this is? Perhaps in the SparseTensor constructor, we can set a new field with the TensorShape.

@reedwm reedwm added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Oct 11, 2017
@ebrevdo
Copy link
Contributor
ebrevdo commented Oct 11, 2017

I'm wary of changing the SparseTensor api. what if you create 3 placeholders and create a SparseTensor from them. does this work?

@reedwm
Copy link
Member
reedwm commented Oct 16, 2017

The API would not change, except that calling get_shape would not prevent feeding the tf.sparse_placeholder. I do not think that making that change would affect backwards compatibility.

3 placeholders would work (that's what tf.sparse_placeholder internally does) but ideally calling get_shape on tf.sparse_placeholder would not cause it to be unfeedable.

@tensorflowbutler
Copy link
Member

It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

1 similar comment
@tensorflowbutler
Copy link
Member

It has been 14 days with no activity and the awaiting tensorflower label was assigned. Please update the label and/or status accordingly.

@tensorflowbutler tensorflowbutler removed the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Jan 23, 2018
@tensorflowbutler
Copy link
Member

A member of the TensorFlow organization has replied after the stat:awaiting tensorflower label was applied.

@tatatodd
Copy link
Contributor

Assigning to @reedwm since he's already analyzed the exact problem, and seems to have a reasonable proposed fix. That said, I don't know the details here.

@reedwm
Copy link
Member
reedwm commented Jan 30, 2018

I don't know enough about SparseTensor to fix this, or currently have time to look into it.

Assigning to @ebrevdo for now.

@reedwm reedwm assigned ebrevdo and unassigned reedwm Jan 30, 2018
@ebrevdo
Copy link
Contributor
ebrevdo commented Jan 30, 2018

You can run

sess.run(x, feed_dict={x.indices: indices, x.values: values})

without breaking shape semantics, right?

@reedwm
Copy link
Member
reedwm commented Jan 30, 2018

Yes, that works, because the shape is not fed.

@tensorflowbutler
Copy link
Member

Nagging Assignee: It has been 14 days with no activity and this issue has an assignee. Please update the label and/or status accordingly.

@ebrevdo
Copy link
Contributor
ebrevdo commented Feb 14, 2018

OK great; closing.

@ebrevdo ebrevdo closed this as completed Feb 14, 2018
@ghost
Copy link
ghost commented Jul 22, 2018

I have the same problem too.
But it seems that no one can solve this.
Any ideas?

@ghost
Copy link
ghost commented Jul 22, 2018

BY the way. tensorflow is really full of bugs itself, which is nothing comparable to pytorch.

@liqimai
Copy link
Author
liqimai commented Jul 22, 2018

@xChenSky
According to my experience, do NOT specify the shape when declare tf.sparse_placeholder.

x = tf.sparse_placeholder(tf.float32) # work
x = tf.sparse_placeholder(tf.float32, shape=shape) # not work

copybara-service bot pushed a commit that referenced this issue Jun 12, 2024
Imported from GitHub PR openxla/xla#13639

Resolves a use-after-free when matching and rewriting layer norm patterns. See #13606.
Copybara import of the project:

--
91ebf7b4a2ac90ebadce27d1a73e88fb4513aed4 by Philipp Hack <phack@nvidia.com>:

Resolves a use-after-free in the norm rewriter.

Merging this change closes #13639

FUTURE_COPYBARA_INTEGRATE_REVIEW=openxla/xla#13639 from philipphack:u_layer_uaf_xla 91ebf7b4a2ac90ebadce27d1a73e88fb4513aed4
PiperOrigin-RevId: 642447222
copybara-service bot pushed a commit that referenced this issue Jun 12, 2024
Imported from GitHub PR openxla/xla#13639

Resolves a use-after-free when matching and rewriting layer norm patterns. See #13606.
Copybara import of the project:

--
91ebf7b4a2ac90ebadce27d1a73e88fb4513aed4 by Philipp Hack <phack@nvidia.com>:

Resolves a use-after-free in the norm rewriter.

Merging this change closes #13639

PiperOrigin-RevId: 642548037
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants