Convert escaping Throwable in copyLogSegmentData to RemoteStorageException - #833
Open
bingkunyangvungle wants to merge 2 commits into
Open
Convert escaping Throwable in copyLogSegmentData to RemoteStorageException#833bingkunyangvungle wants to merge 2 commits into
bingkunyangvungle wants to merge 2 commits into
Conversation
…ption When an Error (e.g. OutOfMemoryError, NoClassDefFoundError) or unchecked Throwable escapes RemoteStorageManager.copyLogSegmentData, it bypasses RLMTask's catch (Exception) on the Kafka side and trips ScheduledThreadPoolExecutor.scheduleWithFixedDelay's silent-suppression contract — all further executions for that partition are dropped, and copy permanently stops until broker restart or leader change. Wrap the body in a try/catch so any escaping Throwable is converted to RemoteStorageException, which the broker's existing catch handles, allowing the scheduled task to retry on the next tick. Refs Aiven-Open#820
…ption Covers four cases via Mockito mockConstruction of KafkaRemoteStorageManager: - OutOfMemoryError -> wrapped as RemoteStorageException (cause preserved) - NoClassDefFoundError -> wrapped as RemoteStorageException (cause preserved) - IllegalStateException -> wrapped as RemoteStorageException (cause preserved) - pre-existing RemoteStorageException -> passes through unchanged (no double-wrap)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes the silent-exit class of failure described in #820.
Problem
If a Throwable escapes
RemoteStorageManager.copyLogSegmentData— for example, anOutOfMemoryErrorfrom inside multipart-upload buffering, aNoClassDefFoundError, or a re-thrownErrorfrom a finally block — it bypasses Kafka's RLMTaskcatch (Exception)clause (becauseErroris not anException) and trips the JDK contract forScheduledThreadPoolExecutor.scheduleWithFixedDelay: "if any execution of the task encounters an exception, subsequent executions are suppressed." The partition's RLMTask permanently stops firing; local on-disk grows monotonically until broker restart or leader change.Fix
Wrap the upload body in a try/catch so any escaping Throwable is converted to
RemoteStorageException. SinceRemoteStorageException extends Exception, the broker's existing catch swallows it and the schedule survives — the next 30s tick allocates a freshRemoteLogSegmentIdand retries.Validation
Reproduced the wedge locally by injecting an
OutOfMemoryErrorinsidecopyLogSegmentDataon the unpatched plugin running against Kafka 3.7: the affected partition stopped uploading and accumulated local segments while sibling partitions continued normally. With this patch applied, the same injection produces a single error log line, the broker's existingcatch (Exception)handles the wrapped exception, and the next scheduled tick succeeds.Closes #820