Can I use S3 server side encryption

[quartox]

Which part is this question about
Library API

Describe your question
I need to use server side encryption when uploading to S3. The upload is rejected without specifying this option, e.g. from aws cli --server-side-encryption AES256. Based on the AmazonS3ConfigKey and ClientKey enums it looks like this is not supported but I am not certain if I understand this correctly. Is there currently a way to specify this argument when uploading?

[tustvold]

There isn't great support for this, but you could possibly use https://docs.rs/object_store/latest/object_store/struct.ClientOptions.html#method.with_default_headers to specify the necessary headers, provided the keys are known at construction time

[wjones127]

I'm looking at this for SSE-KMS support, and it looks like the problem with default header will be that AWS S3 accepts the encryption headers for PUT requests, but will return 400 Bad Request if you pass them along to GET and HEAD requests:

If your object uses SSE-KMS, don't send encryption request headers for GET requests and HEAD requests, or you’ll get an HTTP 400 BadRequest error.¹

So I think these are likely configurations that need to be added to the AmazonS3 store.

Footnotes

1. https://docs.aws.amazon.com/AmazonS3/latest/userguide/specifying-kms-encryption.html ↩

[tustvold]

I plan to work on adding support for this and related use-cases such as #5329 tomorrow

[tustvold]

I think whilst #5334 will potentially provide a way to configure this on a per-operation basis, there is also a use-case for being able to configure this for an entire bucket through properties on the builder.

I think this would be a relatively straightforward addition, and I would be happy to review it

[wjones127]

In keeping with the goals of this crate, I thought I might see if there is a general pattern across object store implementations for how encryption is handled. Indeed there does seem to be a pattern.

In general, there are two modes:

1. Keys are managed by a KMS, and object store can directly reach out and get keys from it. Here, they only need to be provided when writing. On read, the object store can lookup which key it needs to decrypt with from metadata and fetch as needed.
2. Keys are provided by the user, and must be provided on ever write and read request. The object store does not keep the keys, just a digest of them.

For the most part, encryption information is provided as headers as part of requests. The one exception to this is GCS uses query parameters, but only for it's JSON API when in KMS mode. User-provided keys are always passed via headers, and GCP's XML API always uses headers regardless of mode. 😵

For all three object stores, user-provided keys are always AES-256 keys. They are always accompanied by a digest. AWS uses MD5, while Azure and GCP use sha256.

Potential Shared API

This makes me think there might be a general configuration API that could be passed into builders.

// TODO: do not derive debug so encryption keys aren't accidentally printed.
enum EncryptionMode {
  /// Use a key management store
  KMS {
      /// If not provided, will attempt to use a default key, if any.
      key_id: Option<String>,
  },
  /// Provide a key with each request
  Key {
    /// The encryption key to use. Most services require this to be 
    key: Bytes,
    /// The digest of the key. If this is not provided, this will be computed
    /// from the key based on the object stores preferred algorithm, if known.
    /// This is MD5 for AWS S3 and sha256 for others.
    digest: Option<Bytes>,
  }
}

Or perhaps this should be more of a struct like ClientOptions, which is the best parallel I can think of to this situation.

I'll think about this a little more, and likely submit something just implementing AWS for now. But I do think it would be nice to have an API that would work well for other object stores that support server-side encryption.

[tustvold]

GCP's XML API always uses headers regardless of mode.

We only use the XML API

Keys are managed by a KMS

I suspect this is what the vast majority of users are after, and it might be worth starting with just this, unless you specifically require user-provided key support

Potential Shared API

I wonder if we can just lean on the existing configuration system for this, i.e. just allow the different stores to have their own configurations for encryption keys, etc... This isn't materially different from any store specific configuration IMO