KAFKA-20732: seed highestOffsetInRemoteStorage before size-based remote retention [4.1]#22737
Open
horkyada wants to merge 1 commit into
Open
KAFKA-20732: seed highestOffsetInRemoteStorage before size-based remote retention [4.1]#22737horkyada wants to merge 1 commit into
horkyada wants to merge 1 commit into
Conversation
…te retention Backport of apache#22661 (merged to trunk and 4.3.2) to 4.1. On a freshly-elected tiered-partition leader, RLMExpirationTask can run size-retention while highestOffsetInRemoteStorage is still -1 (seeded only by the copy and follower tasks, which race it); onlyLocalLogSegmentsSize then counts the whole local log, double-counted with remoteLogSizeBytes, producing a false retention.bytes breach that deletes in-retention data from both tiers. Mirror the follower path: seed it from remote metadata before size-retention (no-op once seeded). Also widens UnifiedLog.highestOffsetInRemoteStorage() to public, which trunk/4.3 already have and which the guard requires. Regression test ported to the 4.1 harness (RED without the fix, GREEN with it).
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Backport of #22661 to the
4.1branch (already merged to trunk and 4.3.2).Problem
On a freshly elected leader of a tiered partition, size-retention can run while
highestOffsetInRemoteStorageis still-1— it is seeded only by the copy and follower tasks, which race the expiration task.UnifiedLog.onlyLocalLogSegmentsSize()then filtersbaseOffset >= -1and counts the entire local log, which is also counted inremoteLogSizeBytesfor the same offsets. Apparent size roughly doubles, a falseretention.bytesbreach fires, and in-retention data is deleted from the remote tier and — via the resultinglogStartOffsetadvance replicated to followers — the local tier on all replicas. The follower path already guards this (RLMFollowerTask.execute()); the leader/expiration path did not.Fix
In
cleanupExpiredRemoteLogSegments, whenhighestOffsetInRemoteStorage() == -1, seed it from remote metadata (findHighestRemoteOffset) before computing size-retention. No-op once already seeded.Note: this also widens
UnifiedLog.highestOffsetInRemoteStorage()from package-private topublic. trunk and 4.3 already have it public; the== -1guard requires it.Testing
Ported the trunk regression test to 4.1's
RemoteLogManagerTestharness. Verified RED without the fix (both copied segments deleted,logStartOffset -> 200) and GREEN with it; fullRemoteLogManagerTestpasses.