Spark: Remove closing of IO in SerializableTable* #12129
+17
−92
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
I am happy to get input here as to whether or not this is the correct way to solve this issue and am happy to adapt as necessary. Thanks! This effectively reverts: #8924 |
mgmarino
force-pushed
the
fix-removal-of-broadcast-variable
branch
from
cb81b3b to
cac8ceb
Compare
mgmarino
force-pushed
the
fix-removal-of-broadcast-variable
branch
2 times, most recently
from
1855162 to
59c743b
Compare
This is to fix: apache#12046 To summarize, the issue is that Spark can remove broadcast variables from memory and persist them to disk in case that memory needs to be freed. In the case that this happens, the IO object would be closed even if it was still being used by tasks. This fixes the issue by removing the closure of the IO object when the serializable table is closed. The IO objects should be closed on thread finalizers.
mgmarino
force-pushed
the
fix-removal-of-broadcast-variable
branch
from
59c743b to
04137d3
Compare
|
@aokolnychyi Any chance you'll be able to give some feedback on this? Thanks! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This is to fix: #12046
To summarize, the issue is that Spark can remove broadcast variables from memory and
persist them to disk in case that memory needs to be freed. In the case
that this happens, the IO object would be closed even if it was still
being used by tasks.
This fixes the issue by removing the closure of the IO object when the
serializable table is closed. The IO objects should be closed on thread
finalizers.