[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

standalone pip package for tf.io.gfile.GFile #45942

Open
kpe opened this issue Dec 23, 2020 · 7 comments
Open

standalone pip package for tf.io.gfile.GFile #45942

kpe opened this issue Dec 23, 2020 · 7 comments
Labels
comp:ops OPs related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower type:feature Feature requests

Comments

@kpe
Copy link
kpe commented Dec 23, 2020

Is there a chance to somehow split the io.gfile into a standalone pip package? This would be great!

The current tf.io.gfile.GFile API is very handy for transparently accessing local or GCS bucket files.
It is so great, I wish I could use it even in projects where I don't have tensorflow. As far as I'm aware the only/main alternative is to use the google-cloud-storage pip package (google.cloud.storage.Client), but it does not provide a python ideomatic API and cannot transparently handle local files.

I'm not sure where and what would be the best way to handle this, therefore I posted a related issue at googleapis/python-storage#354.
@kpe kpe added the type:feature Feature requests label Dec 23, 2020
@ravikyram ravikyram added the comp:ops OPs related issues label Dec 23, 2020
@ravikyram ravikyram assigned ymodak and unassigned ravikyram Dec 23, 2020
@vnghia
Copy link
Contributor
vnghia commented Dec 23, 2020

@kpe Do you mean something like this https://github.com/dask/gcsfs ?

@kpe
Copy link
Author
kpe commented Dec 23, 2020

@vnvo2409 - thank you, I've missed dask/gcsfs, but it seems to be what I need! Thank you!

(I still like the way how tf.io.gfile.GFile can also open local files, something that seems to not be the case with dask/gcsfs, but it will do)

@vnghia
Copy link
Contributor
vnghia commented Dec 23, 2020

I think it is possible to compile a pip package for gfile without tensorflow but it will require a lot of efforts.

@ymodak ymodak added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Dec 23, 2020
@mihaimaruseac mihaimaruseac self-assigned this Feb 17, 2021
@mihaimaruseac
Copy link
Collaborator

After conversion to modular filesystems fully lands, we might revisit this, as it becomes possible.

@tensorflowbutler tensorflowbutler removed the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Feb 20, 2021
@ymodak ymodak added the stat:awaiting tensorflower Status - Awaiting response from tensorflower label Mar 12, 2021
@ymodak ymodak removed their assignment Mar 12, 2021
@tjulinfan
Copy link

+1 gfile is a beautiful tool which is not only for GCS but also other file systems like HDFS. It will be great we can have it as a standalone package.

@francisco-perez-sorrosal
Copy link

+1 for a standalone package. HDFS access through GFile is nice to have independently of TF

@Corea
Copy link
Contributor
Corea commented Dec 8, 2021

+1

@mihaimaruseac mihaimaruseac removed their assignment Nov 29, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
comp:ops OPs related issues stat:awaiting tensorflower Status - Awaiting response from tensorflower type:feature Feature requests
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
9 participants
@mihaimaruseac @francisco-perez-sorrosal @Corea @kpe @vnghia @tensorflowbutler @ymodak @ravikyram @tjulinfan