Skip to content

Commit 57eba6a

Browse files
HonahXHonahX
committed
Merge branch 'main' into manifest_compaction
2 parents a32471e + 3c225a7 commit 57eba6a

File tree

4 files changed

+28
-1
lines changed

4 files changed

+28
-1
lines changed

‎.github/workflows/check-md-link.yml

+3
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,9 @@ on:
44
push:
55
paths:
66
- mkdocs/**
7+
branches:
8+
- 'main'
9+
pull_request:
710

811
jobs:
912
markdown-link-check:

‎mkdocs/docs/SUMMARY.md

+4
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,8 @@
1717

1818
<!-- prettier-ignore-start -->
1919

20+
<!-- markdown-link-check-disable -->
21+
2022
- [Getting started](index.md)
2123
- [Configuration](configuration.md)
2224
- [CLI](cli.md)
@@ -28,4 +30,6 @@
2830
- [How to release](how-to-release.md)
2931
- [Code Reference](reference/)
3032

33+
<!-- markdown-link-check-enable-->
34+
3135
<!-- prettier-ignore-end -->

‎mkdocs/docs/configuration.md

+20
Original file line numberDiff line numberDiff line change
@@ -81,6 +81,8 @@ For the FileIO there are several configuration options available:
8181

8282
### S3
8383

84+
<!-- markdown-link-check-disable -->
85+
8486
| Key | Example | Description |
8587
| -------------------- | ------------------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
8688
| s3.endpoint | https://10.0.19.25/ | Configure an alternative endpoint of the S3 service for the FileIO to access. This could be used to use S3FileIO with any s3-compatible object storage service that has a different endpoint, or access a private S3 endpoint in a virtual private cloud. |
@@ -91,17 +93,25 @@ For the FileIO there are several configuration options available:
9193
| s3.proxy-uri | http://my.proxy.com:8080 | Configure the proxy server to be used by the FileIO. |
9294
| s3.connect-timeout | 60.0 | Configure socket connection timeout, in seconds. |
9395

96+
<!-- markdown-link-check-enable-->
97+
9498
### HDFS
9599

100+
<!-- markdown-link-check-disable -->
101+
96102
| Key | Example | Description |
97103
| -------------------- | ------------------- | ------------------------------------------------ |
98104
| hdfs.host | https://10.0.19.25/ | Configure the HDFS host to connect to |
99105
| hdfs.port | 9000 | Configure the HDFS port to connect to. |
100106
| hdfs.user | user | Configure the HDFS username used for connection. |
101107
| hdfs.kerberos_ticket | kerberos_ticket | Configure the path to the Kerberos ticket cache. |
102108

109+
<!-- markdown-link-check-enable-->
110+
103111
### Azure Data lake
104112

113+
<!-- markdown-link-check-disable -->
114+
105115
| Key | Example | Description |
106116
| ----------------------- | ----------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
107117
| adlfs.connection-string | AccountName=devstoreaccount1;AccountKey=Eby8vdM02xNOcqF...;BlobEndpoint=http://localhost/ | A [connection string](https://learn.microsoft.com/en-us/azure/storage/common/storage-configure-connection-string). This could be used to use FileIO with any adlfs-compatible object storage service that has a different endpoint (like [azurite](https://github.com/azure/azurite)). |
@@ -112,8 +122,12 @@ For the FileIO there are several configuration options available:
112122
| adlfs.client-id | ad667be4-b811-11ed-afa1-0242ac120002 | The client-id |
113123
| adlfs.client-secret | oCA3R6P\*ka#oa1Sms2J74z... | The client-secret |
114124

125+
<!-- markdown-link-check-enable-->
126+
115127
### Google Cloud Storage
116128

129+
<!-- markdown-link-check-disable -->
130+
117131
| Key | Example | Description |
118132
| -------------------------- | ------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
119133
| gcs.project-id | my-gcp-project | Configure Google Cloud Project for GCS FileIO. |
@@ -128,6 +142,8 @@ For the FileIO there are several configuration options available:
128142
| gcs.default-location | US | Configure the default location where buckets are created, like 'US' or 'EUROPE-WEST3'. |
129143
| gcs.version-aware | False | Configure whether to support object versioning on the GCS bucket. |
130144

145+
<!-- markdown-link-check-enable-->
146+
131147
## REST Catalog
132148

133149
```yaml
@@ -145,6 +161,8 @@ catalog:
145161
cabundle: /absolute/path/to/cabundle.pem
146162
```
147163
164+
<!-- markdown-link-check-disable -->
165+
148166
| Key | Example | Description |
149167
| ---------------------- | ----------------------- | -------------------------------------------------------------------------------------------------- |
150168
| uri | https://rest-catalog/ws | URI identifying the REST Server |
@@ -156,6 +174,8 @@ catalog:
156174
| rest.signing-name | execute-api | The service signing name to use when SigV4 signing a request |
157175
| rest.authorization-url | https://auth-service/cc | Authentication URL to use for client credentials authentication (default: uri + 'v1/oauth/tokens') |
158176

177+
<!-- markdown-link-check-enable-->
178+
159179
### Headers in RESTCatalog
160180

161181
To configure custom headers in RESTCatalog, include them in the catalog properties with the prefix `header.`. This

‎mkdocs/docs/index.md

+1-1
Original file line numberDiff line numberDiff line change
@@ -61,7 +61,7 @@ You either need to install `s3fs`, `adlfs`, `gcsfs`, or `pyarrow` to be able to
6161

6262
## Connecting to a catalog
6363

64-
Iceberg leverages the [catalog to have one centralized place to organize the tables](https://iceberg.apache.org/catalog/). This can be a traditional Hive catalog to store your Iceberg tables next to the rest, a vendor solution like the AWS Glue catalog, or an implementation of Icebergs' own [REST protocol](https://github.com/apache/iceberg/tree/main/open-api). Checkout the [configuration](configuration.md) page to find all the configuration details.
64+
Iceberg leverages the [catalog to have one centralized place to organize the tables](https://iceberg.apache.org/concepts/catalog/). This can be a traditional Hive catalog to store your Iceberg tables next to the rest, a vendor solution like the AWS Glue catalog, or an implementation of Icebergs' own [REST protocol](https://github.com/apache/iceberg/tree/main/open-api). Checkout the [configuration](configuration.md) page to find all the configuration details.
6565

6666
For the sake of demonstration, we'll configure the catalog to use the `SqlCatalog` implementation, which will store information in a local `sqlite` database. We'll also configure the catalog to store data files in the local filesystem instead of an object store. This should not be used in production due to the limited scalability.
6767

0 commit comments

Comments
 (0)