Skip to content

add datafusion-cli support of external table locations that object_store supports #3424

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
kmitchener opened this issue Sep 9, 2022 · 3 comments · Fixed by #3540
Closed
Labels
enhancement New feature or request

Comments

@kmitchener
Copy link
Contributor

Is your feature request related to a problem or challenge? Please describe what you are trying to do.
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and why for this feature, in addition to the what)

Describe the solution you'd like
A clear and concise description of what you want to happen.

Within the datafusion-cli, I'd like to be able to run a create external table SQL with a location like an S3 bucket and have it try to acquire auth from the env.

create external table lineitem stored as parquet location 's3://my_bucket/lineitem/';
-- or 
create external table logs stored as csv partitioned by (year, month) location 's3://my_other_bucket/logs/';
-- or whatever location prefix for the Azure and GCP connection..

This will be useful for testing, as well as for tutorials showing features of DataFusion.

Describe alternatives you've considered
A clear and concise description of any alternative solutions or features you've considered.

Additional context
Add any other context or screenshots about the feature request here.

@kmitchener kmitchener added the enhancement New feature or request label Sep 9, 2022
@turbo1912
Copy link
Contributor

Hi @kmitchener, did some digging on this issue and looks like there is some prior thinking on this. Most specifically this issue.

@kmitchener
Copy link
Contributor Author

@turbo1912 yes, and the object_store is new since then as well. Also, @timvw has just today published some cookbooks regarding object_store and s3 that may help with this: https://github.com/datafusion-contrib/datafusion-cookbook

If you want to give it a shot, go for it!

@timvw
Copy link
Contributor

timvw commented Sep 17, 2022

More inspiration can be found in https://github.com/datafusion-contrib/datafusion-catalogprovider-glue where we take info from the metastore to register external tables...

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

Successfully merging a pull request may close this issue.

3 participants