Put object with a buffer larger than 2 GiB

[ahmedriza]

I notice that if I try to use the put_object API with body that's larger than 2 GiB, the request fails with error writing a body to connection.

Here's a sketch of the code:

    pub async fn put_buffer(
        &self,
        bucket: &str,
        key: &str,
        buffer: Vec<u8>,
    ) -> anyhow::Result<()> {
        let input = ByteStream::new(SdkBody::from(buffer));
        let _ = self
            .client
            .put_object()
            .bucket(bucket)
            .key(key)
            .body(input)
            .send()
            .await?;
        Ok(())
    }

With debug logging enabled, we can see the following (I'm writing the data to a local MinIO instance). The OS is Mac OS Ventura 13.4.1.

[2023-06-24T12:15:07Z DEBUG hyper::client::connect::dns] resolving host="localhost"
[2023-06-24T12:15:07Z DEBUG hyper::client::connect::http] connecting to [::1]:9002
[2023-06-24T12:15:07Z DEBUG hyper::client::connect::http] connected to [::1]:9002
[2023-06-24T12:15:07Z DEBUG hyper::proto::h1::dispatch] error writing: Invalid argument (os error 22)
[2023-06-24T12:15:07Z INFO  aws_smithy_client::poison] poisoning connection: SmithyConnection { is_proxied: false, remote_addr: Some([::1]:9002) }
[2023-06-24T12:15:07Z INFO  aws_smithy_http::connection] smithy connection was poisoned
[2023-06-24T12:15:07Z DEBUG hyper::client::connect] connection was poisoned poison_pill=PoisonPill@0x6000003da9b0 { poisoned: true }
[2023-06-24T12:15:07Z DEBUG aws_smithy_client] send_operation; status="dispatch_failure"
[2023-06-24T12:15:07Z DEBUG aws_smithy_client] send_operation; message=dispatch failure: io error: error writing a body to connection: Invalid argument (os error 22): Invalid argument (os error 22) (DispatchFailure(DispatchFailure { source: ConnectorError { kind: Io, source: hyper::Error(BodyWrite, Os { code: 22, kind: InvalidInput, message: "Invalid argument" }), connection: Unknown } }))

What I can see is that this failure happens when the buffer size is greater than 2 GiB. Is this expected?

Thanks.

[rcoh]

For uploading files this large you should be using mutlipart uploads for better reliability and performance

[kamikaze]

is it possible to stream data somehow? is there any example?