-
Notifications
You must be signed in to change notification settings - Fork 74k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Memory leak in tf.data when iterating over Dataset.from_generator #65675
Comments
Hi Venkat- thanks for checking it out. I’ve only seen the issue in newer
versions of TF with Python 3.11- it seems ok with 3.10. It looks like you
ran it with 3.10 (correct me if I’m mistaken). Can you try with 3.11?
Thanks,
Nick
…On Mon, Apr 15, 2024 at 2:16 AM Venkat6871 ***@***.***> wrote:
Hi ***@***.*** <https://github.com/cohaegen>* ,
I tried to run your code on colab using TF v2.15, 2.16.1, and nightly. But
i am not facing any issue. Please find the gist
<https://colab.sandbox.google.com/gist/Venkat6871/b13bb1502f15e0d47eeb043f96a44be0/65675_2-15-2-16-nightly.ipynb>
here for reference.
Thank you!
—
Reply to this email directly, view it on GitHub
<#65675 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AJBJI645INTV5YBHGK4QYT3Y5OLHDAVCNFSM6AAAAABGGJBR46VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANJWGM2DAMRZGA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Hi @cohaegen ,
Thank you! |
To resurface this, I am also seeing memory leaks when using Python 3.11 and Tensorflow 2.16.1, but it's not with the generator. I'm using the following pipeline:
I haven't spent too much time debugging, but this is what I am facing. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Issue type
Bug
Have you reproduced the bug with TensorFlow Nightly?
Yes
Source
binary
TensorFlow version
v1.12.1-108954-g88310ddcbdd 2.17.0-dev20240412
Custom code
Yes
OS platform and distribution
Docker: tensorflow/tensorflow:nightly
Mobile device
No response
Python version
3.11.0rc1
Bazel version
No response
GCC/compiler version
No response
CUDA/cuDNN version
No response
GPU model and memory
No response
Current behavior?
I discovered what appears to be a memory leak when iterating over a tf.data.Dataset created with from_generator. Process memory usage continues to grow out of hand. The effect only appears in certain combinations of Tensorflow and Python and it may have appeared in Python 3.11. Here are some examples I've tested:
Python 3.10.10, tensorflow 2.13.0: yes
Python 3.10.10, tensorflow 2.16.1: no
Python 3.10.12, tensorflow v2.15.0-0-g6887368d6d4: no
Python 3.11, tensorflow 2.16.1: yes
Python: 3.11.0rc1, tensorflow v1.12.1-109002-g2c2c0a17f05: yes
Maybe related to https://docs.python.org/3/whatsnew/3.11.html#faster-cpython? I thought maybe Python is re-using the memory and not freeing it, but usage grows ridiculously (I noticed it because it started taking up tens of GB in one case) and it seems like it shouldn't with a generator. Odd that 2.13.0 experiences the problem with Python 3.10.10 too though.
Standalone code to reproduce the issue
https://colab.research.google.com/drive/1LmdIqWME19GLFG0E7dsCtRtscLLFF89R?usp=sharing
Relevant log output
The text was updated successfully, but these errors were encountered: