Merge branch 'apache:main' into python-3.12

Author: steinsgateted

.github/workflows/python-release.yml

@@ -80,7 +80,7 @@ jobs:
         if: startsWith(matrix.os, 'ubuntu')
         run: ls -lah dist/* && cp dist/* wheelhouse/
 
-      - uses: actions/upload-artifact@v3
+      - uses: actions/upload-artifact@v4
         with:
           name: "release-${{ github.event.inputs.version }}"
           path: ./wheelhouse/*

.github/workflows/stale.yml

@@ -31,7 +31,7 @@ jobs:
     if: github.repository_owner == 'apache'
     runs-on: ubuntu-22.04
     steps:
-      - uses: actions/stale@v8.0.0
+      - uses: actions/stale@v9.0.0
         with:
           stale-issue-label: 'stale'
           exempt-issue-labels: 'not-stale'

mkdocs/docs/api.md

@@ -33,6 +33,22 @@ catalog:
     credential: t-1234:secret
 ```
 
+Note that multiple catalogs can be defined in the same `.pyiceberg.yaml`:
+
+```yaml
+catalog:
+  hive:
+    uri: thrift://127.0.0.1:9083
+    s3.endpoint: http://127.0.0.1:9000
+    s3.access-key-id: admin
+    s3.secret-access-key: password
+  rest:
+    uri: https://rest-server:8181/
+    warehouse: my-warehouse
+```
+
+and loaded in python by calling `load_catalog(name="hive")` and `load_catalog(name="rest")`.
+
 This information must be placed inside a file called `.pyiceberg.yaml` located either in the `$HOME` or `%USERPROFILE%` directory (depending on whether the operating system is Unix-based or Windows-based, respectively) or in the `$PYICEBERG_HOME` directory (if the corresponding environment variable is set).
 
 For more details on possible configurations refer to the [specific page](https://py.iceberg.apache.org/configuration/).

mkdocs/docs/configuration.md

@@ -74,6 +74,7 @@ For the FileIO there are several configuration options available:
 | s3.signer            | bearer                   | Configure the signature version of the FileIO.                                                                                                                                                                                                            |
 | s3.region            | us-west-2                | Sets the region of the bucket                                                                                                                                                                                                                             |
 | s3.proxy-uri         | http://my.proxy.com:8080 | Configure the proxy server to be used by the FileIO.                                                                                                                                                                                                      |
+| s3.connect-timeout   | 60.0                     | Configure socket connection timeout, in seconds.                                                                                                                                                                                                          |
 
 ### HDFS
 
@@ -140,8 +141,9 @@ catalog:
 
 ## SQL Catalog
 
-The SQL catalog requires a database for its backend. As of now, pyiceberg only supports PostgreSQL through psycopg2.
-The database connection has to be configured using the `uri` property (see SQLAlchemy's [documentation for URL format](https://docs.sqlalchemy.org/en/20/core/engines.html#backend-specific-urls)):
+The SQL catalog requires a database for its backend. PyIceberg supports PostgreSQL and SQLite through psycopg2. The database connection has to be configured using the `uri` property. See SQLAlchemy's [documentation for URL format](https://docs.sqlalchemy.org/en/20/core/engines.html#backend-specific-urls):
+
+For PostgreSQL:
 
 ```yaml
 catalog:
@@ -150,6 +152,22 @@ catalog:
     uri: postgresql+psycopg2://username:password@localhost/mydatabase
 ```
 
+In the case of SQLite:
+
+<!-- prettier-ignore-start -->
+
+!!! warning inline end "Development only"
+    SQLite is not built for concurrency, you should use this catalog for exploratory or development purposes.
+
+<!-- prettier-ignore-end -->
+
+```yaml
+catalog:
+  default:
+    type: sql
+    uri: sqlite:////tmp/pyiceberg.db
+```
+
 ## Hive Catalog
 
 ```yaml

mkdocs/requirements.txt

@@ -23,6 +23,6 @@ mkdocstrings-python==1.7.5
 mkdocs-literate-nav==0.6.1
 mkdocs-autorefs==0.5.0
 mkdocs-gen-files==0.5.0
-mkdocs-material==9.4.14
+mkdocs-material==9.5.3
 mkdocs-material-extensions==1.3.1
 mkdocs-section-index==0.3.8

poetry.lock

@@ -153,7 +153,7 @@ def infer_catalog_type(name: str, catalog_properties: RecursiveDict) -> Optional
                 return CatalogType.REST
             elif uri.startswith("thrift"):
                 return CatalogType.HIVE
-            elif uri.startswith("postgresql"):
+            elif uri.startswith(("sqlite", "postgresql")):
                 return CatalogType.SQL
             else:
                 raise ValueError(f"Could not infer the catalog type from the uri: {uri}")

pyiceberg/catalog/__init__.py

@@ -104,7 +104,7 @@ class GenericDynamoDbError(DynamoDbError):
     pass
 
 
-class CommitFailedException(RESTError):
+class CommitFailedException(Exception):
     """Commit failed, refresh and try again."""
 
 

pyiceberg/exceptions.py

@@ -51,6 +51,7 @@
 S3_SESSION_TOKEN = "s3.session-token"
 S3_REGION = "s3.region"
 S3_PROXY_URI = "s3.proxy-uri"
+S3_CONNECT_TIMEOUT = "s3.connect-timeout"
 HDFS_HOST = "hdfs.host"
 HDFS_PORT = "hdfs.port"
 HDFS_USER = "hdfs.user"

pyiceberg/io/__init__.py

@@ -49,6 +49,7 @@
     GCS_TOKEN,
     GCS_VERSION_AWARE,
     S3_ACCESS_KEY_ID,
+    S3_CONNECT_TIMEOUT,
     S3_ENDPOINT,
     S3_PROXY_URI,
     S3_REGION,
@@ -127,6 +128,9 @@ def _s3(properties: Properties) -> AbstractFileSystem:
     if proxy_uri := properties.get(S3_PROXY_URI):
         config_kwargs["proxies"] = {"http": proxy_uri, "https": proxy_uri}
 
+    if connect_timeout := properties.get(S3_CONNECT_TIMEOUT):
+        config_kwargs["connect_timeout"] = connect_timeout
+
     fs = S3FileSystem(client_kwargs=client_kwargs, config_kwargs=config_kwargs)
 
     for event_name, event_function in register_events.items():