"gossip" - unclear semantic, timing-dependent partitioning, and not really a publish-subscribe service?

[tschudin]

Trying out the gossip example (browser-chat) left me with several questions which I could not resolve by looking into the various explanation pieces:

This iroh repo declares that you can use "iroh-gossip for establishing publish-subscribe overlay networks that scale".

What exactly is the service that the gossip protocol implements? I wonder whether pubSub is the right name for the intended service: wouldn't "multicast group" be the more appropriate name?

Now, assuming it is really pubsub (but the following problem also applies if one would call it "multicast"): this would require that the "topic"/"channel" (or multicast group) lives on, regardless of peers going offline. However, what I observed with the browser-chat example and using the CLI is that when the creator node leaves the channel, new peers can only seemingly join the channel, without becoming visible and without hearing the old peers, while those still connected can continue to use the channel as before.

That is, the creator becomes a kind of server which is what p2p programming wants to overcome, and subscribers find themselves in strange "channel" partitions. It is as if the same gossip channel had several instances (which I would call partitions), despite using the same ticket: If new peers join an orphaned channel (= the creator is currently not online), they can't talk to any of the old peers in that channel.

Where is the life-cycle of a "iroh gossip channel" (or as it seems "channel instances") described? Note that gossip tickets are not explained, either.

Simple sequence of actions to demonstrate the partitioning:

1. Alice creates a channel with ticket T
2. Bob joins T
3. Carla joins T (from a machine far away)
4. all three can chat
5. Alice stops the CLI program
6. Alice joins T
7. Alice cannot talk to Bob and Carla, while Bob and Carla continue to have their channel
8. If Carla stops her CLI program, and joins T again, then a) Alice and Carla are in their own partition while b) Bob is now left fully isolated despite still being online and never having left the channel.

Second observation: It seems that there is a different treatment of local peers vs remote peers, or it is a timing problem. The following case uses the same ticket T that Alice had created above. I made sure that all three peers first went offline and are now reconnecting, in the following sequence:

10. all peers shut down their CLI browser-chat program
11. Bob joins T (on machine X)
12. Carla joins T (on machine Y, far away) - as said above, Bob and Carla cannot talk to each other
13. Alice joins T (on machine X) --> Alice and Bob will be in the same channel, but Carla cannot hear them nor speak to them.

If I disconnect all three peers and immediately afterwards and rapidly execute the steps 11) to 13), then all three can talk to each other. If I wait 5 minutes after disconnection, then steps 11) to 13) even leave Alice and Bob in their own partition and nobody can talk to any other peer. So there seems to be some garbage collection going on in the background? Such semantics is quite difficult to explain and to work with.

As pointed out, "server"-style behavior or "not-yet-garbage-collected-channel-instances-can-be-rejoined-otherwise-its-a-partition" are suboptimal in a p2p and distributed systems context. I can't tell whether this is a limitation of the gossip protocol or due to a buggy implementation. From what I observed, this is neither pubSub nor multicast.

Do I miss something?

[matheus23]

However, what I observed with the browser-chat example and using the CLI is that when the creator node leaves the channel, new peers can only seemingly join the channel, without becoming visible and without hearing the old peers, while those still connected can continue to use the channel as before.

This is because you always need someone from the topic to "introduce you" to others in the topic.

You can re-generate a ticket to join from another node that's still in the topic. That ticket will then have that node's ID set as the bootstrap node.

If the bootstrap node goes offline, you won't be able to join the same topic.

In theory you can generate tickets that contain many bootstrap nodes at the same time, or you can remember the nodes you were in contact with the last time you joined a particular topic (if you went offline in between), or you can use some kind of passive node discovery like our mDNS-based one to get introduced to new NodeIds and try to see if they're also interested in the same topic. It all depends on the application though (I don't think any of this is done by default).

---

Thanks for the feedback on not explaining the gossip ticket. Can I ask you where you initially got introduced to the browser-chat example, so I know where to best add this explanation? Were you coming from the iroh.computer/docs site?

[tschudin]

Can I ask you where you initially got introduced to the browser-chat example, so I know where to best add this explanation?

Such an explanation will certainly help, thanks for your reply.

I went to the iroh-example repo directly (browsing on Github), cloned that and picked the chat example to learn about iroh's gossip service which is never explained, not even in the iroh-gossip repo. The iroh-gossip repo would be the second place which needs that service definition, from what I saw.

If the bootstrap node goes offline, you won't be able to join the same topic.

I see, so it's not pubsub (nor multicast group) and that word should then be removed from, or at least qualified in some way, in the iroh repo. This could be for example "pubsub-like service"?

The requirement to re-acquire a ticket comes as a negative surprise. What it means is that we are finding ourselves back in the rendez-vous problem: having a mutualy agreed topic name was meant to solve that problem and now clients must go again with heuristics for getting hold of the current entry point which may not have any shared history with us. I also think about situations where all participating nodes "go offline" because of some splinternet event or a problem within the n0-computer infrastructure - in which case even rejoin-heuristics do not help and we have to start from scratch, have to establish consensus among the many boostrap node candidates.

I would say that this is a business opportunity for n0-computer: offer persistent channel tickets, or a highly available global directory service, to paying customers. Then all above headaches go away and the caption "peer-2-peer that just works" is true again.