Affected Function / Service
awswrangler.s3._fs.open_s3_object / awswrangler.s3
Description
When using open_s3_object(path, mode="w") or open_s3_object(path, mode="wb"), if 0 bytes are written to the stream prior to close() (for instance, creating an empty marker file, writing b"" or "", or opening and closing without write()), the object is neither created in S3 nor truncated/overwritten if it previously existed.
Minimal Local Reproduction (No AWS credentials needed)
import boto3
import moto
import awswrangler as wr
from awswrangler.s3._fs import open_s3_object
@moto.mock_aws
def test_empty_write():
s3 = boto3.client("s3", region_name="us-east-1")
s3.create_bucket(Bucket="mybucket")
# Attempt to write an empty file
with open_s3_object("s3://mybucket/empty.txt", mode="wb") as f:
pass
# The object does not exist in S3!
assert wr.s3.does_object_exist("s3://mybucket/empty.txt") is True # Fails!
Actual Behavior
wr.s3.does_object_exist("s3://mybucket/empty.txt") returns False.
- If
s3://mybucket/empty.txt already exists with previous data, its contents remain unchanged instead of being truncated to 0 bytes.
Expected Behavior
open_s3_object(path, mode="w" / "wb") should behave like Python's standard open(path, "w" / "wb") file streams: closing the stream after writing 0 bytes should upload/create a 0-byte object in S3 (put_object(Body=b"")).
Root Cause
In _S3ObjectBase.close() inside awswrangler/s3/_fs.py:
if self.writable():
if self._parts_count > 0:
...
elif self._buffer.tell() > 0:
...
put_object(...)
When self._parts_count == 0 and self._buffer.tell() == 0 (0 bytes written), both if self._parts_count > 0 and elif self._buffer.tell() > 0 evaluate to False. As a result, put_object is skipped and no S3 call is made to create or update the object.
Proposed Scope
Update _S3ObjectBase.close() in awswrangler/s3/_fs.py so that when self._parts_count == 0, put_object is called regardless of whether self._buffer.tell() is 0 or >0.
Environment
- OS: Mac / Linux
- Python 3.10+
- AWS SDK for pandas (main branch)
I intend to work on a PR for this issue.
Affected Function / Service
awswrangler.s3._fs.open_s3_object/awswrangler.s3Description
When using
open_s3_object(path, mode="w")oropen_s3_object(path, mode="wb"), if 0 bytes are written to the stream prior toclose()(for instance, creating an empty marker file, writingb""or"", or opening and closing withoutwrite()), the object is neither created in S3 nor truncated/overwritten if it previously existed.Minimal Local Reproduction (No AWS credentials needed)
Actual Behavior
wr.s3.does_object_exist("s3://mybucket/empty.txt")returnsFalse.s3://mybucket/empty.txtalready exists with previous data, its contents remain unchanged instead of being truncated to 0 bytes.Expected Behavior
open_s3_object(path, mode="w" / "wb")should behave like Python's standardopen(path, "w" / "wb")file streams: closing the stream after writing 0 bytes should upload/create a 0-byte object in S3 (put_object(Body=b"")).Root Cause
In
_S3ObjectBase.close()insideawswrangler/s3/_fs.py:When
self._parts_count == 0andself._buffer.tell() == 0(0 bytes written), bothif self._parts_count > 0andelif self._buffer.tell() > 0evaluate toFalse. As a result,put_objectis skipped and no S3 call is made to create or update the object.Proposed Scope
Update
_S3ObjectBase.close()inawswrangler/s3/_fs.pyso that whenself._parts_count == 0,put_objectis called regardless of whetherself._buffer.tell()is 0 or >0.Environment
I intend to work on a PR for this issue.