feat: add multistream http download #1710
Conversation
ischasny
force-pushed
the
ivan/feat-multistream-http-download
branch
2 times, most recently
from
3425fbc to
cc0d047
Compare
There was a problem hiding this comment.
Looks great so far 👍
If I'm reading the code correctly, it will split the file into chunks (5 by default), and make a request for the chunks in parallel. However it will only start reading the response stream for each chunk once the previous chunk has completed. So I'm not sure this will end up being faster than a single stream download.
Great point. I missed the bit that it downloads as it proceeds through the stream. Let me revise the logic and reopen the PR. |
ischasny
force-pushed
the
ivan/feat-multistream-http-download
branch
2 times, most recently
from
b7c6e69 to
93cdde0
Compare
* Add a feature to split http download into multiple chunks. Chunks are downloaded into temporary files, which then get merged into one; * Downnloads over libp2p are always done in one chunk only; * Add NChunks configuration parameter; * Fix tests to handle not just open-ended range requests.
ischasny
force-pushed
the
ivan/feat-multistream-http-download
branch
from
93cdde0 to
46fe02c
Compare
| @@ -119,6 +121,16 @@ func (h *httpTransport) Execute(ctx context.Context, transportInfo []byte, dealI | |||
| } | |||
| tInfo.URL = u.Url | |||
|
|
|||
| // don't allow private network addresses as per https://en.wikipedia.org/wiki/Private_network | |||
There was a problem hiding this comment.
I think we want this to be configurable. We don't really know what kind of setups SPs have and what software they've integrated Boost with... I wouldn't be surprised if some deals are made with private IPs for fetching data. (think of companies that handle both storing and onboarding of data, of which there are many)
|
Splitting this PR into multiple small ones. The first one: #1723 |
Fixes #1593