HTTP retrieval proposal #747
Conversation
There was a problem hiding this comment.
Thank you @hsanjuan, would be extremely nice if we can pull it off with such small set of changes.
Once we have HTTP basics like user-agent, status code metrics, 503/429/Retry-After (details inline), this is worth testing on Rainbow staging (do A/B test with bitswap-only box and bitswap+http).
ps. Whatever we do, HTTP should be opt-in, with a big EXPERIMENTAL warning.
bitswap/network/http_multiaddr.go
Outdated
| return nil, fmt.Errorf("failed to extract host: %w", err) | ||
| } | ||
| host = hostVal | ||
| case "tcp": |
There was a problem hiding this comment.
nit: include "udp" to future proof HTTP/3 deployments?
| case "http", "https": | ||
| schema = comp.Name |
There was a problem hiding this comment.
This may not be enough due to resolved multiaddrs that have explicit /tls/http or even /tls/sni/../
/ip4/..../tcp/../tls/http/ip4/..../tcp/../tls/sni/example.com/http(for example,libp2p.directpeers could announce HTTP capability this way)- here we need to construct
https://example.com(or manually ensure request is sent with correct SNI and Host header)
- here we need to construct
There was a problem hiding this comment.
mm to clarify... why wouldn't libp2p direct use a /dns/<libp2p-direct-hostname>/tcp/.../https address?
There was a problem hiding this comment.
@lidel sorry, is a multiaddr in the form /ip4/..../tcp/.../tls/http valid? does it mean "do not verify certificate?
For things in the form /ip4/..../tcp/../tls/sni/example.com/http... yeah should setup Host header (they have been resolved and we don't want to re-do that).
There was a problem hiding this comment.
@hsanjuan hm.. /ip4/.../tls/http should be valid as-is – some CAs like ZeroSSL.com allow certs for IP instead of DNS name, and they should pass verification just fine
hsanjuan
force-pushed
the
http-retr2
branch
5 times, most recently
from
279d563 to
7e1160b
Compare
This and subsequent commits introduce an httpnet module at what is known as the "bitswap network layer". The bitswap network layer connects bitswap-peers, sends bitswap messages and receives responses. Bitswap messages are basically a wantlist, a list of CIDs that should be sent if available. httpnet does the same, except instead of sending the bitswap message over bitswap, it triggers http requests for the requested blocks. httpnet is a drop-in addon so that we can request blocks over http, and not only via bitswap. As httpnet is a network, it benefits from all existing wantlist management logic. Any http/2 endpoint should benefit from streamlined requests on a single http connection. A router-network ensures that messages are correctly handled by bitswap or by http requests depending on what the peers are advertising. HTTP requests are given priority in the presence of both. Here are some of the httpnet features: * Peers are marked as Connected when they are able to handle http requets. * Peers are marked as Disconnected when http requests fail repeatedly (MaxRetries). * Server errors trigger backoffs preventing more requests to happen to the same url for a period (Retry-After header or configuration value) * We support several urls per peer, meaning a peer can provide alternative http endpoints which are tried based on number of failures or existing cooldowns. * We translate HAVE requests to HTTP-HEAD requests and BLOCK requests to HTTP-GETs * We support cancellations: ongoing or soon to happen requests for a CID can be cancelled using a "cancel" entry in the wantlist. * We record latency information for peers by pinging regularly. * We discriminate between different errors so that we know whether to move to the next block in a wantlist, or to retry with a different url, or to completely abort. * Options to configure user-agent, max retries etc. are supported.
It does not make sense to retry when we got a reply that asks us not to.
This is a proposal to add HTTP retrieval to Boxo. The current state is highly WIP, but I successfully retrieved something over HTTP, so posting to initiate a discussion over the approach and if we want to pursue it until the end.
Approach
The high-level idea is that most of what lives in
bitswap/clientis actually an "exchange" implementation, with the only real "Bitswap" thing being thatbitswap/networksends HAS/GET requests over bitswap-protocol streams. As such, we should be able to complementbitswap/networkwith an HTTP-retrieval implementation which, instead of fetching things over the bitswap protocol, calls HTTP endpoints as indicated by the provider's/httpaddresses entries.Note that conceptually at least, this is not adding HTTP retrieval into bitswap, but promoting most of the bitswap code to be a reference "Exchange" implementation, which is re-usable for different retrieval protocols (bitswap, http...). That is, we would be talking of an "exchange network" component and not a "bitswap network" component. Renames to this extent are still missing.
Implementation
In order to introduce an http-retrieval "exchange network" we need to:
/httpprovider.To this end:
/httpaddresses in the peerstore of the given peer./httpendpoints when handling a WANT.In my tests plugging it to Kubo, the http-network can be used to retrieve content from a gateway over http. 🥳
The main advantange to this approach is that it is relatively clean to incorporate to the codebase, and keeps most of the code untouched, without having to duplicate any of the complex areas.
Challenges
Bitswap places a lot of importance on managing connectivity events to peers. We avoid requesting things from peers that have not signaled connectivity, we clean peers that have disconnected and re-queue things for peers that disconnect. Thus it seems we must support http-connectivity events. When a libp2p peer connects for bitswap, we know that the connection is setup, handshake has been performed and protocol negotiation has happened. For HTTP these things may not exist so we need to define what means "Connected" (i.e. in the case of https it would mean we have completed SSL handshakes).
Apart from that, the question is what are the elements in the current
bitswap/clientstack that do not apply to HTTP (peerqueues, messagequeues, broadcast, wantsending, prioritization etc.)... and why not? What if a peer disconnects from bitswap but not from http or vice-versa? What if Latency is much worse for bitswap than for http? Perhaps this is all logic for the network-router to know how to choose which network to use to send messages.Otherwise perhaps it is not possible to have a satisfactory implementation this way and we need to start thinking what to copy-paste into a separate "http-exchange" (at least the client part).
Related: #608