Doubled buckets after master switch

[Serpentian]

Yesterday I encountered 150 doubled buckets in the cluster and all of them were caused by master changes. The following happened:

1. The cluster is rebalancing buckets
2. Due to high load on storages during rebalancing, they don't respond to cartridge failover's pings, failover constantly changes masters in the cluster
3. Current master sends the bucket, makes it SENT. Right after that the master changes
4. New master still has bucket in ACTIVE state. The replication from old master to the new one either becomes "stopped" (which we had) or it's very slow.
5. We have 2 buckets in ACTIVE state at the same time on different replicaset master

This is dangerous, since a router doesn't know, which bucket is real, so some of the routers will send RW request to one replicaset, other - to another one. As soon as replication is fixed and new master gets the correct state of the bucket, new master either breaks replication, similar to the #573 (if there're RW refs on that bucket), or it will just silently delete the bucket, loosing the data, which it got during that time, when it didn't have correct information about the bucket

It seems, we should teach router to figure out such broken master and don't route requests to them at all.

[Gerold103]

Connected with #412.

[Gerold103]

It seems, we should teach router to figure out such broken master and don't route requests to them at all.

I wouldn't rely on just the router here. Router must not be the essential part of vshard necessary for its correct usage. For instance, assume we fix it in Lua router. Who will fix that in Go router? Who will fix that in any possible cases where people just directly go to the storages and make the refs themselves?

Besides, doesn't seem even possible to fix on the router. Assume that the router sees that the bucket is ACTIVE in one replicaset. And isn't ACTIVE right now on the other one (RECEIVING, for example). Then the router sends request to the first replicaset. But while the request is in fly, the bucket becomes ACTIVE also on the second replicaset. So, even doing a full scan of all replicasets before each router call won't fix the problem.

Needs to be more robust and not dependent on the router.

Below I was trying to brainstorm. And the findings look like we need to redesign the bucket sending and recovery a little bit.

Will it help for each bucket_send() to sync with all replicas after setting the status to SENDING?

Slow, of course. Could also be batched (set multiple buckets to SENDING in one transaction and sync it), but lets look at the logic first. This would fix the specific problem in the ticket, but won't fix the other one.

Imagine a bucket was SENDING on all nodes. Then finally the transmission is done. And the current master makes it SENT. Then it stops being the master. Another node becomes one. It still has the bucket SENDING. On the other replicaset it is RECEIVING.

The other replicaset activates bucket recovery and sees that this bucket is SENT on the old master of the first replicaset (config of the second replicaset is outdated, doesn't know about the new master of the first replicaset). Then it becomes ACTIVE. And now the second replicaset sends it to the third replicaset completely. The bucket is deleted from the second replicaset entirely.

Now the first replicaset's new master with SENDING bucket goes to the second replicaset to check its status and finds no bucket there. Then it becomes ACTIVE on the first replicaset. And we have a duplicate again.

We could try to fix that by forcing the bucket recovery to check if the remote node is still the master, but it won't help much. Firstly, the remote node might still think it is the master. Secondly, while the response is in the network, the node might stop being the master and any shit can happen.

Even synchronous replication doesn't seem helping here much 🤔, AFAICS. Imagine we enable it. Now, the new master surely has all the data of the old one. And the old one can't commit anything. But unless we also enable MVCC, the old master can make a bucket visible as SENT. So the situation above still happens. MVCC + synchro might help, but our state of synchro still suffers from the false-positive split brains. And they won't even be false-positives unless the entire replication becomes synchronous. So I would say this is not an option.

What we seem to need is more strict bucket sending and recovery. Here are a few things which seem necessary:

1. Once a bucket becomes SENDING, it can't be marked SENT until 100% of replicas in this replicaset notice the SENDING. This way we protect from the bucket being fully sent to some other replicaset while still being ACTIVE in the old replicaset.

2. bucket_send() must stop if it sees that the current node is not a master anymore. It doesn't change any guarantees, but it would be unnecessary work to continue the sending.

3. Before making a bucket SENDING, the master must visit all replicas to make sure they don't have RW refs and they aren't thinking of themselves as masters. This way we try to protect from a previous master still running some remaining RW requests but already being switched to a replica (RW refs break replication when master changes #573).

4. Bucket recovery, when going to a remote node, must sync with the other replicas of that replicaset. This way we protect from the case when the second replicaset has an outdated config and goes to the old master for the first replicaset.

5. If bucket recovery can't find the remote bucket, it must make a fullscan of the cluster to try and find it. This way we protect from the case when a bucket was send from rs1 to rs2. Then from rs2 to rs3. And then rs1 can't see it and makes it ACTIVE.

The measures seem quite radical. I was also thinking in the direction of the buckets never being deleted, but instead being stored in some new state like REMOTE. This would allow not to do the point 5 from above.

The point 3 is worrying me, because it is still possible, that a replica was checked not to have RW refs, then it becomes a master, gets an RW ref, but the old master isn't demoted yet, and also makes it SENDING. This I can't see how can be fixed. It is like if we need to persist the rw-lock. Have one more state like PREPARE_TO_SEND or READ_ONLY. Which would be done before SENDING, and would allow to still have some remaining RW refs to linger, but wouldn't let new ones in. Then a bucket sender could just make this state, wait for replication sync, wait for refs to disappear everywhere, and do the sending. If a new master would appear, it would also have to sync before changing the state to any non-READ_ONLY and wouldn't be able to, because it would notice the old master still doing the send.

All this crap needs an RFC, I think. It is much deeper than it looks and than any idea like ignoring or dropping the refs or patching the router.

[Gerold103]

Another vague idea I had for a while is to never delete the buckets from _bucket. Even after a move the bucket entry would stay as a thombstone so any recovery procedure could know where the bucket went to and follow the bucket's path. But I don't see right away how it can help exactly.