[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[WIP] S3 Datastore #5561

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

[WIP] S3 Datastore #5561

wants to merge 1 commit into from

Conversation

hinshun
Copy link
Contributor
@hinshun hinshun commented Oct 4, 2018 •
edited

Based on #5516
Related: #5235, ipfs/notes#214

This is some initial work to get S3 datastore working with IPFS.
Uses fork of go-ds-s3: ipfs/go-ds-s3#7

I have tested this with an AWS S3 bucket. But BloomFilterSize must be set to a high number otherwise the daemon will send too many Get requests to S3.

~/.ipfs/config example
{
    // ...
    "Datastore": {
    "StorageMax": "10GB",
    "StorageGCWatermark": 90,
    "GCPeriod": "1h",
    "Spec": {
      "mounts": [
        {
          "child": {
            "bucket": "<bucket-name>",
            "region": "us-west-2",
            "regionEndpoint": "s3.us-west-2.amazonaws.com",
            "rootDirectory": "",
            "type": "s3ds"
          },
          "mountpoint": "/blocks",
          "name": "s3ds",
          "type": "log"
        },
        {
          "child": {
            "compression": "none",
            "path": "datastore",
            "type": "levelds"
          },
          "mountpoint": "/",
          "prefix": "leveldb.datastore",
          "type": "measure"
        }
      ],
      "type": "mount"
    },
    "HashOnRead": false,
    "BloomFilterSize": 10000000
  }, // ...

@hinshun hinshun requested a review from Kubuxu as a code owner October 4, 2018 05:35
@magik6k magik6k added the status/blocked Unable to be worked further until needs are met label Oct 4, 2018
@magik6k
Copy link
Member
magik6k commented Oct 4, 2018

Need #5187 first

magik6k
magik6k reviewed Oct 4, 2018
View reviewed changes
plugin/plugins/s3ds/s3ds.go Outdated
}, nil
}

type loggingDatastore struct {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There already is a logging datastore: https://github.com/ipfs/go-ipfs/blob/master/repo/fsrepo/datastores.go#L66

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, just starting to learn the codebase.

@magik6k magik6k removed the status/blocked Unable to be worked further until needs are met label Oct 5, 2018
@hinshun
Copy link
Contributor Author
hinshun commented Oct 8, 2018

I attempted to gx install go-ipfs through a ipfs daemon with s3 datastore and it was very slow and got OOM-killed. For smaller loads it seems to work fine but I think we'll need to properly implement go-ds-s3 with aws-sdk-go.

@MichaelMure
Copy link
Contributor

Any idea if this would allow to run multiple IPFS nodes with a shared s3 datastore ?

@magik6k
Copy link
Member
magik6k commented Oct 19, 2018

If you only use this for /blocks/ and disable GC on all nodes then it may be possible

@hinshun
Copy link
Contributor Author
hinshun commented Oct 19, 2018

As an update, I rewrote ipfs/go-ds-s3 with aws-sdk-go here and was successful in gx installing all of go-ipfs's dependencies.

@hinshun hinshun force-pushed the feat/s3ds-plugin branch 3 times, most recently from b93f21b to b66a91b Compare October 30, 2018 17:16
@hinshun
s3ds plugin
1526a4a 
License: MIT
Signed-off-by: Edgar Lee <edgarl@netflix.com>
@bonedaddy
Copy link
Contributor
bonedaddy commented Feb 15, 2019 •
edited

I've been trying something similar to this and have noticed a critical issue, which is that if you pin some content, and run ipfs pin ls you can see the content as being pinned. If you shutdown the node, start it up and run ipfs pin ls again, nothing shows up.

@momack2 momack2 added this to Inbox in ipfs/go-ipfs May 9, 2019
@georgyo
Copy link
georgyo commented Oct 1, 2020

Should this ticket be closed in favor of making ipfs/go-ds-s3 in the default plugin list?

@guseggert guseggert mentioned this pull request Mar 4, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@magik6k magik6k magik6k left review comments

@Kubuxu Kubuxu Awaiting requested review from Kubuxu

At least 1 approving review is required to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
No open projects
ipfs/go-ipfs
Inbox
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
5 participants
@hinshun @magik6k @MichaelMure @bonedaddy @georgyo