Skip to content

bug: wr.s3.open_s3_object in write mode does not create or overwrite 0-byte objects #3421

Description

@hsusul

Affected Function / Service

awswrangler.s3._fs.open_s3_object / awswrangler.s3

Description

When using open_s3_object(path, mode="w") or open_s3_object(path, mode="wb"), if 0 bytes are written to the stream prior to close() (for instance, creating an empty marker file, writing b"" or "", or opening and closing without write()), the object is neither created in S3 nor truncated/overwritten if it previously existed.

Minimal Local Reproduction (No AWS credentials needed)

import boto3
import moto
import awswrangler as wr
from awswrangler.s3._fs import open_s3_object

@moto.mock_aws
def test_empty_write():
    s3 = boto3.client("s3", region_name="us-east-1")
    s3.create_bucket(Bucket="mybucket")

    # Attempt to write an empty file
    with open_s3_object("s3://mybucket/empty.txt", mode="wb") as f:
        pass

    # The object does not exist in S3!
    assert wr.s3.does_object_exist("s3://mybucket/empty.txt") is True  # Fails!

Actual Behavior

  • wr.s3.does_object_exist("s3://mybucket/empty.txt") returns False.
  • If s3://mybucket/empty.txt already exists with previous data, its contents remain unchanged instead of being truncated to 0 bytes.

Expected Behavior

open_s3_object(path, mode="w" / "wb") should behave like Python's standard open(path, "w" / "wb") file streams: closing the stream after writing 0 bytes should upload/create a 0-byte object in S3 (put_object(Body=b"")).

Root Cause

In _S3ObjectBase.close() inside awswrangler/s3/_fs.py:

if self.writable():
    if self._parts_count > 0:
        ...
    elif self._buffer.tell() > 0:
        ...
        put_object(...)

When self._parts_count == 0 and self._buffer.tell() == 0 (0 bytes written), both if self._parts_count > 0 and elif self._buffer.tell() > 0 evaluate to False. As a result, put_object is skipped and no S3 call is made to create or update the object.

Proposed Scope

Update _S3ObjectBase.close() in awswrangler/s3/_fs.py so that when self._parts_count == 0, put_object is called regardless of whether self._buffer.tell() is 0 or >0.

Environment

  • OS: Mac / Linux
  • Python 3.10+
  • AWS SDK for pandas (main branch)

I intend to work on a PR for this issue.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions