Add object store support #78
Merged
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Basic Object store support
Changes:
register_parquet,register_listing_table, and addedregister_csvsuch that they will automatically register an object store based on the Url of the path provided. Local paths, relative or absolute, are interpreted as afile://url, using the parse functionality inListingTableUrlfromdatafusion.example/http_csv.pya port ofhttps://github.com/apache/datafusion/blob/45.0.0/datafusion- examples/examples/query-http-csv.rsReadme.mdto be a working example using a github hosted csv file.Object store credentials
In this PR, all object stores are automatically registered - which can be really nice - and the trade off is that we rely on the cloud provider SDKs to gather credentials from their environment, or local files, rather than explicitly providing them.
In a future release we can add more configurability here.
Looks like a few formatting changes snuck in here from editing. I would like to defer sorting out those details until we have commit hooks to ensure proper and consistent file formatting.