Add optional_len to FSReadRequest #13350
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Summary
Context: We want to enable background compaction read sizes to be tuned to the particular storage backend. In this case, we want as many reads as possible to be aligned to the file system block boundaries.
Compaction prefetches go through
FilePrefetchBuffer
. IfFilePrefetchBuffer
had access to the exact block boundaries within the file, it could easily issue optimally sized reads.Indeed, you could expose a
GetBlockSize
orGetNextBlockOffset
API insideRandomAccessFileReader
(and all other child classes). If all block sizes were fixed,GetBlockSize
would giveFilePrefetchBuffer
all the information it needs. If there was some variation,GetNextBlockOffset
would suffice.Those solutions would not be too complicated, but I think we can simplify the solution even further. I think all we really need is an
optional_read_size
field inFSReadRequest
.What does this give us?
RandomAccessFileReader
and its descendants do not need to implement new API methods.FSReadRequest::len
is a valid result size.optional_read_size
. We can enable it only for compaction reads with non-direct IO.Potential concerns:
offset
orlen
, leaving it in an inconsistent state withoptional_read_size
? I was worried about this, and I thought about the scenario where the request got broken up into smaller pieces, butoptional_read_size
was left maintained. This would be terrible obviously because thenoptional_read_size
could exceed the individual smaller request sizes. I think we are OK becauseFSReadRequest
has its copy constructor implicitly deleted because offs_scratch
. Thus, any downstream use cases would have initializedFSReadRequest
and set its parameters from scratch. It is OK for downstream classes to ignoreoptional_read_size
. We are mainly concerned aboutoptional_read_size
getting copied over by accident, and luckily we are already protected against that.Test Plan
Will need to update unit tests inside
FilePrefetchBuffer
. Will create a mock test file system implementation for a file system that does supportoptional_read_size
.