My understanding from #8807 is that there's some analysis being done to decide on the magic number here.

In particular is there any analysis on the amount of added network traffic as a result of reducing the number here and should we expose this number in the config file?

If I'm reading the graph it correctly #8807 (comment) it looks like almost no peers respond in under 100ms and some large fraction (80%?) respond by 1 second.

Dropping to 0ms, 20ms, or anything that's before Bitswap responses come back means we're sending out more requests than we do today. Understanding how many extra requests we're making vs how much time we're saving seems useful. This could also be helpful in assisting users who want to tweak the values between keeping the network chatter lower and waiting longer on Bitswap responses (e.g. gateways vs IPFS Desktop kinds of settings).

There will certainly be more chatter, but if the values at the end of #8807 (comment) are any valid, it's going to be a very small fraction. According to the comment, we're talking about 20k/12million requests (~0.1%), when Bitswap comes back positive. Even if that value is off by a factor of 10, we're still talking about 1% of cases where we make two requests, when one would be enough. But we're also saving minimum one RTT (cases where we receive DONT_HAVE) and maximum 1s (timeout) in 99% of cases.

Overall, my view is that if we start in parallel (or almost in parallel) we don't have much to lose. We can also spin up a node (or a few nodes, not connected to each other) and do the measurement study in the real network when this is merged.

The node(s) would:

• start connections in the network, as they would normally.
• listen for CIDs requested through Bitswap.
• ask for the same CIDs at regular time intervals thereafter, excluding asking the peers where they received the request from (presumably the peer that asked for a CID will eventually get it, so we want to avoid counting false positives).

Has anyone corroborated that 0.1% number? Seems like a pretty critical measurement. The degree to which we make unnecessary reqs is the degree to which we'd further increase the load on DHT servers, which AFAIU is so high that many folks run in client-only mode, so I'm really loathe to further increase DHT server load.

@mrd0ll4r this relates to your previous comment here: #8807 (comment) - any easy way you can instrument this measurement (success/failure of Bitswap discovery) from the infrastructure you already have in place? :)

Hmm, we are working on a paper where we'll be able to do this, but I don't think that's going to happen in the near future. The methodology will be the same as I used for the comment, i.e., manually sending stuff on Bitswap and collecting responses. So no, I can't get this from the passive measurements we're doing at the moment, sorry...

EDIT: also, importantly, we explicitly do not use the usual request mechanism, which uses this timeout. We send stuff on Bitswap directly, without ever triggering a DHT search. So I won't be able to test the impact of this.

About the 20k positive responses/12 million requests number: Keep in mind that

1. I had to read those off the graph, they're not accurately from the dataset,
2. I can't ascertain the popularity of the content I tested with in a larger context, i.e., I don't know if other content is spread much more widely, generating more positive responses (or the opposite), and
3. Those 20k requests were the positive ones, but we did receive a response to almost all requests we sent out. This is because we set the send_dont_have flag, in order to also receive negative responses. This, critically, allows us to not rely on timeouts. (I had a number about what portion of peers respect send_dont_have, but I don't have it at hand. I believe it was the vast majority.)
   I suppose something like that would be possible here, too (although that's another issue): set send_dont_have for all requests, and start searching the DHT as soon as some portion of our neighbors responded without any positive response.

In general, I agree with 20ms being too low as a timeout, because it's unlikely that any meaningful number of responses will have been received. It effectively behaves as 0ms, i.e., do DHT and Bitswap discovery in parallel. If that's the goal, setting it to zero seems less confusing.

I can't ascertain the popularity of the content I tested with in a larger context, i.e., I don't know if other content is spread much more widely, generating more positive responses (or the opposite).

Totally. Some data processing required here, but I expect that ipfs.io gets a lot of requests for data stored with popular pinning services, some of which they're peered with (more than 0.1%).

This knob here deals with resource consumption vs performance, with the added caveat that the resource consumption is both on the DHT server nodes and on the client node itself.

Let's say we take the view that we're not concerned about additional DHT server load (idk that that's a good idea, but let's say). How does a user figure out what they should set the magic number to? Have we tried running this in places like ipfs.io to see what seems optimal at least for that deployment?

While we're figuring this out it seems pretty reasonable to expose a knob in the Bitswap.Internal config, although it should come with some guidance around tuning it.

As an aside as we introduce additional routing systems (e.g. Reframe delegated routers which are already supported) we'll likely need to figure out other ways to handle this. For example, maybe it makes sense to wait 500ms before a DHT request but 0s for a delegated routing request to cid.contact?

There's probably a more ideal dynamic system here than hard coding in some numbers, but if there are real performance benefits to be gained we can hard code some numbers and encourage some follow-up work on more dynamic/adaptable systems here.

Added ProviderSearchDelay to Internal.Bitswap config but left the default at the low value of 20ms. We could restore that to 1s and allow experiments to test the effect of changing it before enshrining a new default in code.

Would that ^ help get results from more existing nodes @guillaumemichel?

As a note, we are starting the Bitswap Discovery Success Rate study this week and expect to have first results in the next couple of weeks. Work is being tracked here: https://pl-strflt.notion.site/Effectiveness-of-Bitswap-Discovery-Process-aab515ebc1774bee8193da415bfb98f4 - any input more than welcome.

For now can we expose this config but leave the default alone until at least we review @guillaumemichel's results?

Yes. Will change to 1s

Can you revert this to 1 second please ? Let try having this as an option before changing defaults.