remove find_missing_blobs deduplication to fix bug #1121
Closed
+26
−107
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Previously, as part of an optimization to FindMissingBlobs calls, the find_missing_cache was changed so that the first task to ask about a blob could notify others of the result when it was finished. This introduced a bug related to cancellation: If the first task is canceled before it finishes, every other task that was waiting on the result fails with a "channel closed" error. Since then, there has been a GetDigestsTtlDeduper introduced at a higher layer, outside of the remote execution client, that deduplicates these same calls. This change completely gets rid of attempting to deduplicate FindMissingBlobs calls inside of the remote execution client. At the moment, I think this is the best course of action, and to get deduplication of these calls, it should be preferred to enable deduplicate_get_digests_ttl_calls.
meta-cla
bot
added
the
CLA Signed
|
@facebook-github-bot has imported this pull request. If you are a Meta employee, you can view this in D84729147. (Because this pull request was imported automatically, there will not be any future comments.) |
|
Looks good. I want to reference a comment from @JakobDegen from the original diff that introduced GetDigestsTtlDeduper.
If doing this across the board is something that would benefit you, we can certainly consider it. |
|
This pull request has been merged in 040ce41. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Previously, as part of an optimization to FindMissingBlobs calls, the find_missing_cache was changed so that the first task to ask about a blob could notify others of the result when it was finished. This introduced a bug related to cancellation: If the first task is canceled before it finishes, every other task that was waiting on the result fails with a "channel closed" error.
Since then, there has been a GetDigestsTtlDeduper introduced at a higher layer, outside of the remote execution client, that deduplicates these same calls.
This change completely gets rid of attempting to deduplicate FindMissingBlobs calls inside of the remote execution client. At the moment, I think this is the best course of action, and to get deduplication of these calls, it should be preferred to enable deduplicate_get_digests_ttl_calls.