Make s3.request_timeout configurable #1568

metadaddy · 2025-01-23T22:37:06Z

Similarly to #218, we see occasional timeout errors when writing data to S3-compatible object storage:

When uploading part for key 'drivestats/data/date_month=2014-08/00000-0-9c7baab5-af18-4558-ae10-1678aa90b6a5.parquet' in bucket 'drivestats-iceberg': AWS Error NETWORK_CONNECTION during UploadPart operation: curlCode: 28, Timeout was reached

[I don't believe the issue is specific to the fact that I'm using Backblaze B2 rather than Amazon S3 - I saw references to similar error messages with the latter as I was researching this issue.]

The issue happens when the underlying PUT operation takes longer than the request timeout, which is set to a default of 3 seconds in the AWS C++ SDK used by Arrow via PyArrow.

The changes in this PR allow configuration of s3.request_timeout when working directly or indirectly with pyiceberg.io.pyarrow.PyArrowFileIO, just as #218 allowed configuration of s3.connect_timeout.

For example, when creating a catalog:

catalog = load_catalog(
    "docs",
    **{
        "uri": "http://127.0.0.1:8181",
        "s3.endpoint": "http://127.0.0.1:9000",
        "py-io-impl": "pyiceberg.io.pyarrow.PyArrowFileIO",
        "s3.access-key-id": "admin",
        "s3.secret-access-key": "password",
        "s3.request-timeout": 5.0,
        "s3.connect-timeout": 20.0,
    }
)

Fokko

Thanks @metadaddy for adding this, I left one comment regarding S3FS, apart from that it looks good to me 👍

Fokko · 2025-01-24T12:48:35Z

pyiceberg/io/fsspec.py

@@ -150,6 +151,9 @@ def _s3(properties: Properties) -> AbstractFileSystem:
    if connect_timeout := properties.get(S3_CONNECT_TIMEOUT):
        config_kwargs["connect_timeout"] = float(connect_timeout)

+    if request_timeout := properties.get(S3_REQUEST_TIMEOUT):
+        config_kwargs["request_timeout"] = float(request_timeout)


I think this should be read_timeout, looking at: https://github.com/fsspec/s3fs/blob/51e3c80ef380a82081a171de652e2b699753be2b/s3fs/core.py#L473-L479

@Fokko You're entirely correct - thanks for catching that! I'll make the correction and push the updated code.

Fokko · 2025-01-24T12:49:31Z

pyiceberg/io/pyarrow.py

@@ -394,6 +395,9 @@ def _initialize_oss_fs(self) -> FileSystem:
        if connect_timeout := self.properties.get(S3_CONNECT_TIMEOUT):
            client_kwargs["connect_timeout"] = float(connect_timeout)

+        if request_timeout := self.properties.get(S3_REQUEST_TIMEOUT):
+            client_kwargs["request_timeout"] = float(request_timeout)


Looks good: https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html

Fokko · 2025-01-24T12:50:29Z

pyiceberg/io/pyarrow.py

+        if request_timeout := self.properties.get(S3_REQUEST_TIMEOUT):
+            client_kwargs["request_timeout"] = float(request_timeout)


Looks good: https://arrow.apache.org/docs/python/generated/pyarrow.fs.S3FileSystem.html

Fokko · 2025-01-24T12:52:24Z

mkdocs/docs/configuration.md

@@ -116,6 +116,7 @@ For the FileIO there are several configuration options available:
 | s3.region            | us-west-2                  | Configure the default region used to initialize an `S3FileSystem`. `PyArrowFileIO` attempts to automatically resolve the region for each S3 bucket, falling back to this value if resolution fails.                                                                                                           |
 | s3.proxy-uri         | <http://my.proxy.com:8080> | Configure the proxy server to be used by the FileIO.                                                                                                                                                                                                                    |
 | s3.connect-timeout   | 60.0                       | Configure socket connection timeout, in seconds.                                                                                                                                                                                                                        |
+| s3.request-timeout   | 60.0                       | Configure socket read timeouts on Windows and macOS, in seconds.                                                                                                                                                                                                        |


I couldn't find a Java equivalent, so I'm fine with introducing this one 👍

metadaddy · 2025-01-24T18:07:15Z

Hi @Fokko - I implemented and pushed your suggested correction. Thanks!

kevinjqliu · 2025-01-24T19:11:51Z

Looks like theres a lint issue, can you make make lint locally? @metadaddy

Fokko reviewed Jan 24, 2025

View reviewed changes

add request timeout config

87fcad5

metadaddy force-pushed the fs_request_timeout branch from 1675f74 to 87fcad5 Compare January 24, 2025 18:05

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make s3.request_timeout configurable #1568

Make s3.request_timeout configurable #1568

metadaddy commented Jan 23, 2025

Fokko left a comment

Fokko Jan 24, 2025

metadaddy Jan 24, 2025

Fokko Jan 24, 2025

Fokko Jan 24, 2025

Fokko Jan 24, 2025

metadaddy commented Jan 24, 2025 •

edited

Loading

kevinjqliu commented Jan 24, 2025

		if request_timeout := self.properties.get(S3_REQUEST_TIMEOUT):
		client_kwargs["request_timeout"] = float(request_timeout)

Make s3.request_timeout configurable #1568

Are you sure you want to change the base?

Make s3.request_timeout configurable #1568

Conversation

metadaddy commented Jan 23, 2025

Fokko left a comment

Choose a reason for hiding this comment

Fokko Jan 24, 2025

Choose a reason for hiding this comment

metadaddy Jan 24, 2025

Choose a reason for hiding this comment

Fokko Jan 24, 2025

Choose a reason for hiding this comment

Fokko Jan 24, 2025

Choose a reason for hiding this comment

Fokko Jan 24, 2025

Choose a reason for hiding this comment

metadaddy commented Jan 24, 2025 • edited Loading

kevinjqliu commented Jan 24, 2025

metadaddy commented Jan 24, 2025 •

edited

Loading