Fix timeouts not working when not LEADING or FOLLOWING #2057
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Details
When a server is neither LEADING nor FOLLOWING we don't check for command timeouts. Normally this doesn't matter, because a server should become LEADING or FOLLOWING very soon, but we run into this when shutting down clusters in tests. If you are the last node left in a cluster, you won't have quorum, and thus cannot process commands.
This is why we've changed BedrockTester to take a timeout in the same PR, because we do this in auth:
https://github.com/Expensify/Auth/blob/0751f4eefd49945a009cc219152fe3b51a684502/test/test/AuthTester.cpp#L219
Because when this is done via
Query
command (as it is for HC-Tree) we can send aQuery
to a node that is never able to process it, and the node will never shut down.I think in production, this is relatively unlikely to occur, but I still think it's "more correct" to honor timeouts in this state.
There are some particulars here about not having a DB handle when we do this, which is generally required to timeout commands because we attach a
CommitCount
message to even timed out command responses. This skips attaching that line in this particular case, and changes how some parameters are passed/stored/etc inBedrockCore
to accomodate not always having anSQLite
object available.Fixed Issues
Found while working on https://github.com/Expensify/Expensify/issues/337537
Tests
Existing
Internal Testing Reminder: when changing bedrock, please compile auth against your new changes