provider: Expose provider connectivity status

[phillebaba]

The new provider is a great addition and removes a lot of complexity burden for projects using the DHT, amazing work!

After doing some experiments with the provider I realized that it would be nice to expose some of the connectivity functionality outside of the package. Specifically the isOffline function would be nice to be able to call to be able to use in health checking.

Additionally I was a bit inspired by the internal connectivity package that checks if the router is able to find other peers. My thinking here is that I want to implement some custom bootstrap triggering when the provider is in an offline state. While I could probably implement this on my own outside the provider package it would mean that the checking logic would have to be run twice. Would it be possible to either additional callback handlers for online and offline events, alternatively the ability to define my own connectivity implementation.

[guillaumemichel]

Specifically the isOffline function would be nice to be able to call to be able to use in health checking.

Since #1144 is merged, you can access the connectivity status by calling prov.Stats().Connectivity.Status, see

go-libp2p-kad-dht/provider/stats.go

Lines 36 to 50 in 095fe97

	// Connectivity status
	var status string
	if s.connectivity.IsOnline() {
	status = "online"
	} else {
	if avgPrefixLenCached >= 0 {
	status = "disconnected"
	} else {
	status = "offline"
	}
	}
	snapshot.Connectivity = stats.Connectivity{
	Status: status,
	Since: s.connectivity.LastStateChange(),
	}

Calling prov.Stats() generates some work (computing moving averages of other metrics), so calling it too often wouldn't be ideal to check the connectivity state of the system.

Would it be possible to either additional callback handlers for online and offline events, alternatively the ability to define my own connectivity implementation.

It seems like the best solution would be to:

• Make the connectivity package public (e.g move provider/internal/connectivity to provider/connectivity)
• Define an interface along with the default implementation
• Allow users to provide their own connectivity checker when building a SweepingProvider (e.g WithConnectivity option)
  • If using custom connectivity checker, SweepingProvider must be able to add its own callback functions (must be part of the interface)

This would allow:

• User defined callbacks on connectivity status change
• User can check connectivity status, without extending the SweepingProvider interface

@phillebaba would that satisfy your needs?

[phillebaba]

What would the interface be for the connectivity function? It would require isOnline and then the callbacks for online and offline events.

I ended up implementing my own logic similar to connectivity in spegel-org/spegel#1051 mostly to understand what would work well for me. I ended up completely removing returning bootstrap peers from a function and implementing similar functionality on my own. Mostly because my use case is a bit different than IPFS and the bootstrap nodes are semi deterministically found by sorting round robin DNS record lookups.

In my use cases having access to isOnline and a trigger for when offline is all I really need in the end. While the trigger is less important it would be nice to not have two tickers checking the same health.

Out of interest is there a reason that GetClosestPeers is used for the health checking? Other than the fact that the router does not expose access to the routing table?

[guillaumemichel]

What would the interface be for the connectivity function?

The interface could look like this

type ConnectivityChecker interface {
  Start()
  Close() error
  AddCallbacks(onOnline, onDisconnected, onOffline []func())
  TriggerCheck()
  IsOnline() bool
  LastStateChange() time.Time
}

The main change would be AddCallbacks() would replace the current SetCallbacks(). AddCallbacks() could be called multiple times before Start() is called. Callbacks would be executed in the order they are added, and user cannot reorder them.

Then, the SweepingProvider could accept a WithConnectivityChecker() option, and add its own callbacks to it. In your usecase, you could create an instance of the default ConnectivityChecker (or make your own implementation of the interface), add your callbacks and then use it to build the SweepingProvider.

Note that you can track the connectivity state (online, disconnected, offline) using custom callbacks.

---

Out of interest is there a reason that GetClosestPeers is used for the health checking? Other than the fact that the router does not expose access to the routing table?

For any provide operation, the SweepingProvider needs to find the closest peers to allocate the provider records. It does so by calling GetClosestPeers(), so if the function doesn't work (because bootstrappers missing, peer offline, dht issues) then the reprovider cannot perform any operation, so it shouldn't even try.

The router doesn't expose access to the routing table because the provider interface aims to remain generic, and doesn't strictly depends on the go-libp2p-kad-dht IpfsDHT. Users just need to provide a way to find the DHT closest peers.

[phillebaba]

Interface looks great, this would cover all of my needs to trigger bootstrap once the state changes to offline. Also makes it easy to indicate to Kubernetes probes that the instance is not healthy.

Makes sense that it uses GetClosestPeers as a health check, instead of checking the routing table size. Thanks for the insight!

Is there anything I can do to help on this work?

[hsanjuan]

Triage:

• @guillaumemichel will try to address it when he has time