PS-10416 / PS-11509: Send Content-MD5 header for S3 uploads to support Object Lock - #42
Open
kamil-holubicki wants to merge 1 commit into
Open
PS-10416 / PS-11509: Send Content-MD5 header for S3 uploads to support Object Lock#42kamil-holubicki wants to merge 1 commit into
kamil-holubicki wants to merge 1 commit into
Conversation
…t Object Lock https://perconadev.atlassian.net/browse/PS-10416 https://perconadev.atlassian.net/browse/PS-11509 Problem: util.dumpInstance() to an AWS S3 bucket with Object Lock enabled fails because S3 requires a client-side checksum on every payload-bearing PUT that creates a new object version. Without Content-MD5 (or an x-amz-checksum-* header) both single-shot PutObject and multipart UploadPart are rejected with 400 'Content-MD5 OR x-amz-checksum-* HTTP header is required ... with Object Lock parameters'. Cause: Neither PutObject nor UploadPart was emitting a client-side checksum header. Small dumps that fit in a single-shot PutObject and larger dumps that go through the multipart UploadPart path were failing for the same underlying reason. Solution: Inject Content-MD5 once, at signing time, inside the AWS request signer. Every payload-bearing S3 request funnels through that signer - it is where the SigV4 authorization header and the x-amz-content-sha256 payload hash are already assembled - so a single addition there covers both PutObject and UploadPart without touching any bucket, container, or backend class. The injection is guarded so it only fires when the request carries a client-side body, leaving no-body operations (HeadObject, GetObject, DeleteObject, CopyObject, UploadPartCopy, CreateBucket) untouched, and it preserves any Content-MD5 the caller already set - notably the one that S3_bucket::delete_objects computes over its batch-delete XML body. The signature-caching layer already bypasses the cache whenever the request carries a body, so the MD5 is recomputed fresh whenever the payload changes. Anonymous access short-circuits earlier and bypasses signing; that is intentional and safe because S3 rejects anonymous PUT/UploadPart against an Object-Lock bucket as unauthorized before Object Lock is evaluated.
|
Hi, thank you for your contribution. Please confirm this code is submitted under the terms of the OCA (Oracle's Contribution Agreement) you have previously signed by cutting and pasting the following text as a comment: |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
https://bugs.mysql.com/bug.php?id=119568
Problem:
util.dumpInstance() to an AWS S3 bucket with Object Lock enabled fails because S3 requires a client-side checksum on every payload-bearing PUT that creates a new object version. Without Content-MD5 (or an x-amz-checksum-* header) both single-shot PutObject and multipart UploadPart are rejected with 400 'Content-MD5 OR x-amz-checksum-* HTTP header is required ... with Object Lock parameters'.
Cause:
Neither PutObject nor UploadPart was emitting a client-side checksum header. Small dumps that fit in a single-shot PutObject and larger dumps that go through the multipart UploadPart path were failing for the same underlying reason.
Solution:
Inject Content-MD5 once, at signing time, inside the AWS request signer. Every payload-bearing S3 request funnels through that signer - it is where the SigV4 authorization header and the x-amz-content-sha256 payload hash are already assembled - so a single addition there covers both PutObject and UploadPart without touching any bucket, container, or backend class.
The injection is guarded so it only fires when the request carries a client-side body, leaving no-body operations (HeadObject, GetObject, DeleteObject, CopyObject, UploadPartCopy, CreateBucket) untouched, and it preserves any Content-MD5 the caller already set - notably the one that S3_bucket::delete_objects computes over its batch-delete XML body. The signature-caching layer already bypasses the cache whenever the request carries a body, so the MD5 is recomputed fresh whenever the payload changes.
Anonymous access short-circuits earlier and bypasses signing; that is intentional and safe because S3 rejects anonymous PUT/UploadPart against an Object-Lock bucket as unauthorized before Object Lock is evaluated.