Experimental - service discovery

[philhofer]

This doesn't work yet. Right now it doesn't draw a distinction between locally- vs. globally-routable addresses. Unix sockets and loopback addresses should only be cached at distance=0.

[philhofer]

Hey @ttacon -- let me know if you think this is going totally the wrong direction.

There are a number of things missing:

• Deletion propogation (service "blah" is no longer up).
• Periodic re-synchronization. Right now it just happens on connect.
• Automatic route selection (if we connect to a server over link-local, and it connects to another service over link-local on the same interface, then we should be able to reach that service over link-local as well.)
• A "beacon" daemon/tool.

... and probably much more.

[ttacon]

So I still haven't had time to really sit down and look at this as much as I'd like to but I have some initial feelings.

This is something I REALLY want. Having said that, I'm somewhat torn as to whether this belongs in synapse or if it belongs at level higher. Using synapse to build a system which does all of these wonderful things you listed means that synapse doesn't have to deal with bloat - it can stay a far more eloquent version of net/rpc. But, that means we don't have the ability to do any of the nice (and far faster) things we can do if synapse handles the rest of this functionality. I'm just unsure if this is a can of worms we want to open.

With that said, I like the idea of this in synapse. I think it's worth it. I also think if we were to get the first two items on the checklist in, that's a good enough version 1 (of this functionality, not synapse). Because, as you know my feelings on Erlang, I've experienced first hand the joy of cluster partitioning that can come from name synchronization only on connect.

If we agree to go this way, I think first we should decide which features we want this to support and then decide on an iteration plan before hacking away too much. What are your thoughts?

[philhofer]

Yep. Agreed on all fronts. The scope of this feature needs to be better defined. Developing some kind of (questionably robust) distributed consensus algorithm is explicitly a non-goal.

One of the problems that we can solve (that I don't think is typically addressed in ordinary service discovery frameworks) is service "locality." In almost all cases, we'd prefer to connect to external services over unix sockets before TCP, over private networks before public ones, etc. Solving this problem dynamically makes configuration much simpler.

So, what this boils down to solving is: given a known good address to connect to service "foo," can I find a similarly-capable peer that would provide greater service quality? (Of course, the definition of "quality" is up for interpretation. In the example above, it was the reduction of network distance and concomitant risk of network partitions, but it could also mean superior load balancing, or avoiding services that want to shut down cleanly. I'm envisioning that at some point we'll have a way for servers to ask clients to pick a different peer.)

In some ways, this is sort of like solving the congestion control and reliability problems that TCP solved in L3 network protocols, except this time we're solving it (in a small way) by providing distributed multi-homing for L4 protocols.