[go: nahoru, domu]
Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

fix: update soft block limit to 2MiB #8968

Open
wants to merge 1 commit into
base: master
Choose a base branch
from
Open

fix: update soft block limit to 2MiB #8968

wants to merge 1 commit into from

Conversation

Jorropo
Copy link
Contributor
@Jorropo Jorropo commented May 12, 2022

This is needed because the 1MiB limit is the chunker limit, however the
actual blocks can be wrapped in protobuf which adds a small ~10 byte
overhead.

@Jorropo Jorropo requested a review from aschmahmann May 12, 2022 17:06
@Jorropo Jorropo self-assigned this May 12, 2022
@Jorropo Jorropo enabled auto-merge (rebase) May 12, 2022 17:07
@Jorropo Jorropo disabled auto-merge May 12, 2022 17:19
@Jorropo Jorropo force-pushed the fix-big-blocks branch 3 times, most recently from e518f3a to 53d219d Compare May 24, 2022 17:19
@Jorropo Jorropo added this to the go-ipfs 0.13 milestone May 25, 2022
@Jorropo
Copy link
Contributor Author
Jorropo commented May 26, 2022 •
edited

may 26 2022 discussion:
Need to add two tests:

  • ipfs block get $(ipfs add --raw-leaves=false --chunker=size-$(( 1024 * 1024 )) 1MiBFile) | ipfs block put works. (non raw leaves blocks with 1MiB chunks include a ~10 bytes protobuf wrapping making them slightly too big, thoses blocks MUST pass, this is what this PR fixes)
  • top down blackbox exchange test of 2MiB blocks.

@Jorropo
Copy link
Contributor Author
Jorropo commented May 26, 2022

Other possible solution if this drag on too much, revert 04e7e95 in 0.13.0 and move it to 0.14.0.

@BigLep BigLep modified the milestones: go-ipfs 0.13, go-ipfs 0.14 Jun 2, 2022
@lidel lidel modified the milestones: kubo 0.14, kubo 0.15 Jul 20, 2022
lidel
lidel requested changes Jul 20, 2022
View reviewed changes
Copy link
Member
@lidel lidel left a comment •
edited

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm moving this to 0.15: this needs end-to-end regression tests so we are not surprised when something changes deep in one of libp2p deps. We want tests for two cases:

  • confirm two nodes exchange 2MiB block
  • confirm two nodes are unable to exchange 2MiB+1 byte block

@BigLep
Copy link
Contributor
BigLep commented Sep 1, 2022

What's the priority of this? What is motivating this work?

@BigLep BigLep removed the request for review from aschmahmann September 1, 2022 01:52
@Jorropo
Copy link
Contributor Author
Jorropo commented Sep 1, 2022

What is motivating this work?

  1. There is edge cases when you use 1MiB without --raw-leaves you exceed 1MiB because the protobuf wrapper.
  2. linux2ipfs use 2MiB raw leaves blocks and I don't like being annoyed when I use it.

@lidel
Copy link
Member
lidel commented Sep 1, 2022

This still needs tests described in #8968 (review)

@lidel lidel removed this from the kubo 0.15 milestone Sep 1, 2022
@Jorropo
Copy link
Contributor Author
Jorropo commented Sep 1, 2022

I know, I didn't took time to wrote them yet.

@BigLep BigLep marked this pull request as draft September 2, 2022 00:53
@lidel lidel mentioned this pull request Sep 5, 2022
Merged
17 tasks
@Jorropo Jorropo removed the need_tests label Jan 5, 2023
@Jorropo Jorropo marked this pull request as ready for review January 5, 2023 05:53
@Jorropo Jorropo requested a review from lidel January 5, 2023 05:53
@Jorropo
Copy link
Contributor Author
Jorropo commented Jan 5, 2023 •
edited

confirm two nodes are unable to exchange 2MiB+1 byte block

I didnt do this because we don't really have consensus on this: ipfs/specs#331 (comment)

@Jorropo
fix: update soft block limit to 2MiB
efc758a 
This is needed because the 1MiB limit is the chunker limit, however the
actual blocks can be wrapped in protobuf which adds a small ~10 byte
overhead.
@Jorropo Jorropo removed their assignment Mar 4, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@lidel lidel Awaiting requested review from lidel

Requested changes must be addressed to merge this pull request.

Assignees
No one assigned
Labels
None yet
Projects
No open projects
IPFS Shipyard Team
Status: No status
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

None yet
3 participants
@Jorropo @BigLep @lidel