Why does PING discovery fail *but* MPING works?

[mattgagliardi]

I have a situation that's not making much sense (to me anyway). I'm setting up a vanilla Keycloak cluster on a set of machines (not containers...KC 18.0.1 in Quarkus JVM mode, not using Wildfly). When attempting to get the Infinispan cache running the UDP option does not work...OK, I figure no multicasting allowed (that's what IT says). But on a lark I attempted the "cache-stack=tcp" option from the Keycloak manual (as I read it this still relies on multicast for discovery) and the group is formed and Infinispan is happy. The "why does this work?" is killing me. Keycloak looks to be pulling its Infinispan setup from here (link); the very obvious difference between the -udp and -tcp options (besides tcp and udp, etc.) is the discovery protocols. And since that seems to be where this is failing I figure it's got to be that.
I really want to stick with the nice "automatic" discovery of a network-based option rather than moving to JDBC_PING, but before I feel safe doing that I need to understand this. Can anyone offer some thoughts or perspective on what might be happening? TIA!

Quick note: the KC/Infinispan distribution does appear to be using JGroups 4.2 rather than 5.2 as linked above. Sorry.

[mattgagliardi]

Let's close this. Turns out with the benefit of max'd out logging the group is correctly forming regardless of udp or tcp. However in the udp option my Infinispan caches look to have issues staying up-to-date, the log contains messages along the lines of:

2022-07-07 16:51:03,121 DEBUG [org.jgroups.protocols.FD_ALL] (Timer runner-2,srv-kcdoc-1-19975) srv-kcdoc-1-19975: haven't received a heartbeat from srv-kcdoc-0-61660 for 10223 ms, adding it to suspect list
2022-07-07 16:51:03,121 DEBUG [org.jgroups.protocols.FD_ALL] (Timer runner-2,srv-kcdoc-1-19975) srv-kcdoc-1-19975: suspecting [srv-kcdoc-0-61660]
2022-07-07 16:51:03,121 DEBUG [org.jgroups.protocols.FD_ALL] (jgroups-15,srv-kcdoc-1-19975) srv-kcdoc-1-19975: unsuspecting srv-kcdoc-0-61660
2022-07-07 16:51:03,121 DEBUG [org.jgroups.protocols.FD_SOCK] (jgroups-15,srv-kcdoc-1-19975) srv-kcdoc-1-19975: broadcasting unsuspect(srv-kcdoc-0-61660)
2022-07-07 16:51:03,121 DEBUG [org.jgroups.protocols.FD_SOCK] (jgroups-15,srv-kcdoc-1-19975) srv-kcdoc-1-19975: broadcasting unsuspect(srv-kcdoc-0-61660)

That's from srv-kcdoc-1. Guessing(?) the way MPING is asymmetric (link) and transitions from multicast over to TCP/unicast is what's making the difference on my network.

[rhusar]

It does sound that multicast setup in your network is broken rather than "not allowed".

That's from srv-kcdoc-1. Guessing(?) the way MPING is asymmetric (link) and transitions from multicast over to TCP/unicast is what's making the difference on my network.

I guess if you inspect the logs closely from both configurations, the asymmetricity could become apparent. Though – at this point I would recommend sticking with TCP-based stacks.

I really want to stick with the nice "automatic" discovery

There are other ways to do dynamic discovery, as long there is a single shared resource, e.g. a FILE_PING http://www.jgroups.org/manual/html/protlist.html#FILE_PING on a shared (NFS, samba) drive.

[mattgagliardi]

I suspect you’re correct and that for safety’s sake I will have to move to one of these other options. I’m leaning towards JDBC_PING because I can’t guarantee the presence of a shared drive but I’ll definitely always have a database.

[rhusar]

In that case JDBC_PING sounds like an ideal fit.

[dereed]

If multicast doesn't work at all then MPING won't work.
It's more likely (and quite common) that multicast is mostly broken, but not completely blocked.

So you're basically just getting lucky if it happens to work correctly for MPING some of the time. But will most likely not work correctly other times.

[mattgagliardi]

Thanks!